2024-03-25T17:36:09.663 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:36:09.663 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/orig.config.yaml 2024-03-25T17:36:09.702 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:36:09.781 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621299' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC8ogJIpy1TBt+IUhnJ7ZyvgSe3upjMvybdFt9GMEHJsjODab+wJCwQcVzxQfKN8/0bekXnyckYIX/8utXEO7xo= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdJt5xwrd3mWp6ij5wkj+fiGIV0KjKbkFyUqxxutZYyHZkvAiIrk7XMRh7jrWfGA9SGzmoXz15JMe8waoLYMJY= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:36:09.781 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:36:09.782 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:36:09.782 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:36:09.783 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:36:09.804 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:36:09.804 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:36:09.804 INFO:teuthology.packaging:ref: None 2024-03-25T17:36:09.804 INFO:teuthology.packaging:tag: None 2024-03-25T17:36:09.804 INFO:teuthology.packaging:branch: main 2024-03-25T17:36:09.804 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:36:09.804 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:36:10.052 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:36:10.053 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:36:10.060 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:36:10.060 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:36:10.069 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:36:10.084 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:36:10.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:36:10.119 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:45.898802', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC8ogJIpy1TBt+IUhnJ7ZyvgSe3upjMvybdFt9GMEHJsjODab+wJCwQcVzxQfKN8/0bekXnyckYIX/8utXEO7xo='} 2024-03-25T17:36:10.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:45.897102', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdJt5xwrd3mWp6ij5wkj+fiGIV0KjKbkFyUqxxutZYyHZkvAiIrk7XMRh7jrWfGA9SGzmoXz15JMe8waoLYMJY='} 2024-03-25T17:36:10.143 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:36:10.152 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:36:10.152 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:36:10.152 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:36:10.221 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe4a217a0d0>, signals=[15]) 2024-03-25T17:36:10.222 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:36:10.230 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:36:10.230 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T17:36:10.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:36:10.337 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T17:36:10.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:36:10.421 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:36:10.428 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-03-25T17:36:10.447 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-03-25T17:36:10.447 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:36:10.504 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-03-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:36:10.505 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:36:10.505 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-03-25T17:36:10.524 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-03-25T17:36:10.542 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-03-25T17:36:10.542 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-03-25T17:36:10.598 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:36:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:36:10.600 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-03-25T17:36:10.623 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:36:10.634 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:36:10.643 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:36:10.644 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:36:10.646 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:36:10.662 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:36:10.670 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:36:10.670 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:36:10.703 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:36:10.730 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:36:10.785 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:36:10.799 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:36:10.799 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:36:10.799 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:36:10.799 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:36:10.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:36:10.816 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-25T17:36:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:36:10.817 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-03-25T17:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:36:10.873 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-361.el9.x86_64 2024-03-25T17:36:10.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-03-25T17:36:10.874 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:36:10.874 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-361.el9.x86_64 2024-03-25T17:36:10.874 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-03-25T17:36:12.999 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T17:36:13.636 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 9.6 MB/s | 6.7 MB 00:00 2024-03-25T17:36:14.425 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-25T17:36:15.027 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-25T17:36:45.321 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 270 kB/s | 8.0 MB 00:30 2024-03-25T17:36:48.011 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 248 kB/s | 8.0 MB 00:33 2024-03-25T17:36:53.775 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 19 MB 00:07 2024-03-25T17:36:56.884 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 19 MB 00:07 2024-03-25T17:37:05.522 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 782 kB/s | 6.1 MB 00:08 2024-03-25T17:37:07.169 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T17:37:08.306 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 789 kB/s | 6.1 MB 00:07 2024-03-25T17:37:09.127 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T17:37:10.121 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-25T17:37:11.709 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T17:37:14.136 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T17:37:16.401 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:37:16.402 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:37:16.489 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:37:16.501 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:37:16.501 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 76 M 2024-03-25T17:37:16.502 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 123 M 2024-03-25T17:37:16.503 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:37:16.657 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T17:37:17.107 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.131 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.158 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.431 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.452 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.479 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:17.580 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:17.599 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:17.600 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:17.619 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:17.619 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:17.637 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:17.644 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:17.663 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:17.718 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:17.741 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:17.741 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:17.759 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:17.770 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:17.820 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:17.828 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:17.854 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:17.873 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:17.919 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.921 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:17.939 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:18.061 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:18.071 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:18.093 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:18.096 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:18.106 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:18.128 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:18.162 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.170 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.193 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.301 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.311 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.335 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:18.344 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:18.350 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:18.415 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:18.435 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:18.435 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:18.454 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:18.595 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.596 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.615 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.806 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:37:18.806 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:37:18.919 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T17:37:18.924 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T17:37:18.925 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:37:18.925 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:37:18.925 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:37:18.925 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-03-25T17:37:18.925 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:37:18.925 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 76 M 2024-03-25T17:37:18.926 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 123 M 2024-03-25T17:37:18.926 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:37:18.947 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:18.967 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:18.970 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:19.005 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:19.017 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:19.026 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:19.079 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.080 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:19.084 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.084 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.115 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.157 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:19.157 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:19.173 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.196 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.197 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.214 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.215 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.218 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:19.232 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.237 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.237 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.249 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:19.291 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:19.307 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.307 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:19.324 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:19.328 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.328 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.349 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:19.441 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:19.478 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:19.479 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:19.604 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 22 kB/s | 66 kB 00:02 2024-03-25T17:37:19.893 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:19.919 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:19.941 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:20.049 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:20.204 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.2 MB/s | 15 MB 00:03 2024-03-25T17:37:20.261 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:20.295 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:20.305 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:20.339 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:20.374 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:20.405 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:20.483 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:20.503 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.559 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:20.560 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.567 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:20.579 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.581 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.587 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.667 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.677 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:20.684 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.689 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.697 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.704 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 8.2 MB/s | 33 MB 00:04 2024-03-25T17:37:20.773 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.777 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:20.782 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:20.840 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:20.849 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:20.853 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:20.945 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:20.956 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:20.981 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:20.992 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:21.004 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:21.015 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:21.020 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:21.023 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:21.167 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:21.211 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:21.228 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:21.230 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:21.247 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:21.249 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:21.250 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:21.271 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:21.276 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:21.296 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:21.434 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:21.435 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:21.440 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:21.453 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:21.528 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:21.601 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:21.635 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:21.676 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:21.800 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:21.825 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:21.841 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:21.856 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:21.864 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:21.881 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:21.940 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:22.002 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:22.008 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:22.017 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:22.036 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:22.108 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:22.141 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:22.185 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:22.203 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:22.210 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:22.273 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:22.332 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:22.337 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:22.341 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:22.405 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:22.420 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:22.425 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:22.452 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:22.462 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:22.575 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:22.582 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:22.732 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:22.769 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:22.806 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:22.811 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:22.844 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:22.859 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 17 kB/s | 66 kB 00:03 2024-03-25T17:37:22.859 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:22.927 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:22.987 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:23.050 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:23.081 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:37:23.169 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:23.368 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 3.4 MB/s | 15 MB 00:04 2024-03-25T17:37:23.372 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:37:23.808 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:24.129 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.1 MB/s | 27 MB 00:04 2024-03-25T17:37:24.134 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:24.134 INFO:teuthology.orchestra.run.smithi060.stdout:Total 10 MB/s | 76 MB 00:07 2024-03-25T17:37:24.376 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.3 MB/s | 33 MB 00:05 2024-03-25T17:37:24.635 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:37:24.769 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:24.826 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:37:24.838 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:37:24.898 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:24.918 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:24.994 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:25.017 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.114.130.31) 2024-03-25T17:37:25.159 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:25.225 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:37:25.297 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:37:25.297 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:37:25.397 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:25.432 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:37:25.607 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:25.626 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:37:25.660 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:26.049 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:37:26.210 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:26.287 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:37:26.317 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:37:26.317 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:37:26.341 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:37:26.429 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:37:26.505 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:37:27.318 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.2 MB/s | 27 MB 00:04 2024-03-25T17:37:27.323 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:27.323 INFO:teuthology.orchestra.run.smithi090.stdout:Total 9.0 MB/s | 76 MB 00:08 2024-03-25T17:37:27.988 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:37:28.344 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:37:28.344 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:37:28.912 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:37:29.355 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:37:29.355 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:37:29.557 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:37:29.674 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:31.925 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:37:32.548 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:32.622 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:37:32.684 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:32.732 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:37:35.669 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:37.798 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:37.844 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:37:40.349 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:37:40.429 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:37:42.968 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:37:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:13.754 INFO:teuthology.orchestra.run.smithi060.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:38:13.754 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:38:13.802 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:14.190 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:14.190 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:38:14.191 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:38:14.191 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:14.623 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:38:14.624 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:38:14.984 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:15.082 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:15.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:15.083 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:38:15.083 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:38:15.083 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:38:15.083 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:38:15.083 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:38:15.084 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:15.167 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:15.167 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:15.167 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 68.el9 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-03-25T17:38:15.205 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8264254 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-03-25T17:38:15.206 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:38:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:38:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:38:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:38:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-03-25T17:38:15.207 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:38:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:38:15.208 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:38:15.208 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:38:15.208 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:38:15.410 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:15.410 INFO:teuthology.orchestra.run.smithi090.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:38:15.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:38:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:15.931 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-03-25T17:38:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T17:38:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T17:38:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T17:38:16.359 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:38:16.360 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:38:16.750 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:16.882 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:16.882 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:16.882 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T17:38:16.882 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:38:16.882 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:38:16.882 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:38:16.883 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:38:16.883 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:38:16.962 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T17:38:16.963 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T17:38:16.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-03-25T17:38:16.996 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-03-25T17:38:16.996 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-03-25T17:38:16.996 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-03-25T17:38:16.996 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 68.el9 2024-03-25T17:38:16.996 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-03-25T17:38:16.996 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8264254 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:38:16.997 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:38:16.998 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:38:16.998 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-03-25T17:38:16.998 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:38:16.998 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:38:16.998 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:38:16.999 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-432.el9.x86_64 2024-03-25T17:38:16.999 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:38:17.412 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:38:17.433 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-03-25T17:38:17.441 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-03-25T17:38:17.456 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.sd3z0MurAp 2024-03-25T17:38:17.456 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sd3z0MurAp 2024-03-25T17:38:17.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.sd3z0MurAp 2024-03-25T17:38:17.673 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.sd3z0MurAp is 6KB 2024-03-25T17:38:17.722 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.sd3z0MurAp 2024-03-25T17:38:17.736 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:38:17.747 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-03-25T17:38:17.806 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:38:17.806 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:38:17.806 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:38:17.806 INFO:teuthology.orchestra.run.smithi060.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:38:17.808 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:38:17.968 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-03-25T17:38:19.539 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:38:19.560 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-03-25T17:38:19.562 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-03-25T17:38:19.601 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.cz4u2cjJvl 2024-03-25T17:38:19.602 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cz4u2cjJvl 2024-03-25T17:38:19.670 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.cz4u2cjJvl 2024-03-25T17:38:19.826 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.cz4u2cjJvl is 6KB 2024-03-25T17:38:19.877 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.cz4u2cjJvl 2024-03-25T17:38:19.891 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:38:19.955 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:38:19.955 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:38:19.955 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:38:19.955 INFO:teuthology.orchestra.run.smithi090.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T17:38:19.957 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T17:38:20.117 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-03-25T17:38:47.992 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:38:47.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-25T17:38:47.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:38:50.122 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:38:50.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-25T17:38:50.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:06.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.60 2024-03-25T17:39:08.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-25T17:39:15.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-25T17:39:15.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:17.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-25T17:39:17.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:39:20.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-03-25T17:39:32.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-25T17:39:32.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:40:15.444 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:40:27.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-25T17:40:27.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:40:27.729 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-03-25T17:40:28.114 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-25T17:40:28.115 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:40:28.115 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-03-25T17:40:28.130 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:40:28.130 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:40:28.130 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:40:28.130 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:40:29.131 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:40:29.131 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:40:29.204 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-03-25T17:40:29.236 DEBUG:teuthology.parallel:result is None 2024-03-25T17:40:32.525 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T17:40:47.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-25T17:40:47.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:40:48.028 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-25T17:40:48.463 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-25T17:40:48.464 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T17:40:48.464 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-03-25T17:40:48.481 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-432.el9.x86_64 2024-03-25T17:40:48.481 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T17:40:48.481 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:40:48.481 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:40:49.482 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:40:49.482 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:40:49.575 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-03-25T17:40:49.623 DEBUG:teuthology.parallel:result is None 2024-03-25T17:40:49.623 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:40:49.633 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:40:49.633 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:40:49.636 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:40:49.652 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:40:49.659 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:40:49.669 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:40:49.669 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:40:49.693 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:40:49.723 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:40:49.729 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:40:49.729 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:40:49.757 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:40:49.791 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:49.795 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:49.805 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:49.809 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:40:49.811 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:40:49.818 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:40:49.818 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:40:49.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:40:49.880 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:40:49.889 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:40:49.889 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:40:49.919 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:40:49.936 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:50.016 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:50.064 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:50.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:50.188 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T17:40:50.188 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:40:50.256 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:50.300 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:40:50.357 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:50.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:40:50.487 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T17:40:50.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:40:50.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-03-25T17:40:50.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-03-25T17:40:50.602 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:40:50.649 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:40:50.923 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:40:50.931 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:40:50.931 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:40:50.940 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:40:50.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-03-25T17:40:51.005 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-03-25T17:40:51.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-03-25T17:40:51.094 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-03-25T17:40:51.096 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:40:51.096 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-03-25T17:40:51.115 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-03-25T17:40:51.115 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-03-25T17:40:51.138 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-03-25T17:40:51.138 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:40:51.139 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:40:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:40:51.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:40:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:40:51.211 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:40:51.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:40:51.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:40:51.305 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:40:51.315 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:40:51.332 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:40:51.396 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:45:40.221 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2024-03-25T17:45:40.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-25T17:45:40.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:45:40.315 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-03-25T17:45:40.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-03-25T17:45:40.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-25T17:45:40.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:45:40.486 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-03-25T17:45:40.560 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-03-25T17:45:40.561 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:45:40.575 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:45:40.575 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:45:40.576 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:45:40.578 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:45:40.578 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:45:40.607 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:45:40.623 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:45:40.630 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:45:40.649 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:45:40.652 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-03-25T17:45:40.666 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-03-25T17:45:40.676 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-03-25T17:45:40.682 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:45:40.690 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-03-25T17:45:40.699 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:45:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:45:40.722 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:45:40.750 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:45:40.754 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:45:40.754 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-03-25T17:45:40.754 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.754 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.755 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.755 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.774 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:45:40.777 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:45:40.777 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-03-25T17:45:40.778 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.778 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.778 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.778 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:45:40.778 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:45:40.786 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:45:40.787 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:45:40.787 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:45:40.787 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:45:40.792 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:45:40.793 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:45:40.793 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:45:40.793 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:45:40.796 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:45:40.796 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:45:41.023 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:45:41.023 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:45:41.027 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:45:41.027 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:45:41.365 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:45:41.365 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T17:45:41.365 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:45:41.394 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:45:41.394 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:45:41.428 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:45:41.428 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T17:45:41.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:45:41.457 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:45:41.457 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:45:41.461 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:45:41.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:45:41.542 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:45:41.605 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:45:41.609 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-03-25T17:45:41.610 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-03-25T17:45:41.671 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-03-25T17:45:41.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-03-25T17:45:41.860 INFO:teuthology.orchestra.run.smithi060.stdout:74 files removed 2024-03-25T17:45:41.887 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw 2024-03-25T17:45:41.915 INFO:teuthology.orchestra.run.smithi090.stdout:74 files removed 2024-03-25T17:45:41.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-03-25T17:45:43.310 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 73 kB/s | 82 kB 00:01 2024-03-25T17:45:43.457 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 67 kB/s | 82 kB 00:01 2024-03-25T17:45:44.172 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T17:45:44.291 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T17:45:44.953 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-25T17:45:45.117 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T17:45:45.505 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T17:45:45.535 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T17:45:46.995 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-25T17:45:47.064 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-25T17:45:55.687 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 996 kB/s | 8.0 MB 00:08 2024-03-25T17:45:55.688 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 985 kB/s | 8.0 MB 00:08 2024-03-25T17:46:05.641 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:08 2024-03-25T17:46:06.477 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 19 MB 00:09 2024-03-25T17:46:18.111 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 715 kB/s | 6.1 MB 00:08 2024-03-25T17:46:18.368 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 750 kB/s | 6.1 MB 00:08 2024-03-25T17:46:19.567 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-03-25T17:46:19.757 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-03-25T17:46:21.694 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T17:46:26.785 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-25T17:46:29.341 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:46:29.355 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:46:29.355 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:46:29.355 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:46:29.355 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:46:29.355 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:46:29.355 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-03-25T17:46:29.355 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:46:29.356 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:46:29.357 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout:Install 30 Packages 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 2 Packages 2024-03-25T17:46:29.358 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:46:29.359 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 58 M 2024-03-25T17:46:29.359 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:46:40.188 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-25T17:46:45.110 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-25T17:46:46.552 INFO:teuthology.orchestra.run.smithi060.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 325 kB/s | 5.3 MB 00:16 2024-03-25T17:46:46.985 INFO:teuthology.orchestra.run.smithi060.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 58 kB/s | 25 kB 00:00 2024-03-25T17:46:47.602 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:46:47.616 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:46:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:46:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:46:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout:Install 30 Packages 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:46:47.620 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 58 M 2024-03-25T17:46:47.621 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:46:49.511 INFO:teuthology.orchestra.run.smithi060.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 294 kB/s | 742 kB 00:02 2024-03-25T17:46:50.869 INFO:teuthology.orchestra.run.smithi060.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 367 kB/s | 498 kB 00:01 2024-03-25T17:47:01.036 INFO:teuthology.orchestra.run.smithi060.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 318 kB/s | 9.7 MB 00:31 2024-03-25T17:47:01.252 INFO:teuthology.orchestra.run.smithi060.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 209 kB/s | 45 kB 00:00 2024-03-25T17:47:01.652 INFO:teuthology.orchestra.run.smithi060.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 328 kB/s | 131 kB 00:00 2024-03-25T17:47:02.069 INFO:teuthology.orchestra.run.smithi060.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 389 kB/s | 162 kB 00:00 2024-03-25T17:47:02.853 INFO:teuthology.orchestra.run.smithi060.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 409 kB/s | 320 kB 00:00 2024-03-25T17:47:03.745 INFO:teuthology.orchestra.run.smithi060.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 334 kB/s | 298 kB 00:00 2024-03-25T17:47:04.145 INFO:teuthology.orchestra.run.smithi060.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 248 kB/s | 99 kB 00:00 2024-03-25T17:47:04.262 INFO:teuthology.orchestra.run.smithi060.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 911 kB/s | 106 kB 00:00 2024-03-25T17:47:04.303 INFO:teuthology.orchestra.run.smithi060.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-25T17:47:04.337 INFO:teuthology.orchestra.run.smithi060.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T17:47:04.396 INFO:teuthology.orchestra.run.smithi060.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T17:47:04.438 INFO:teuthology.orchestra.run.smithi060.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-25T17:47:04.471 INFO:teuthology.orchestra.run.smithi060.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T17:47:04.512 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:04.603 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:04.727 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:04.748 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:04.890 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:04.966 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:04.998 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:05.054 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:05.213 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:05.397 INFO:teuthology.orchestra.run.smithi060.stdout:(18/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 359 kB/s | 5.1 MB 00:14 2024-03-25T17:47:05.417 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:05.437 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:05.461 INFO:teuthology.orchestra.run.smithi090.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 309 kB/s | 5.3 MB 00:17 2024-03-25T17:47:05.474 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:05.493 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:05.568 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:05.641 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:05.653 INFO:teuthology.orchestra.run.smithi090.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 131 kB/s | 25 kB 00:00 2024-03-25T17:47:05.729 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:05.748 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:05.838 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:06.001 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:06.049 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:06.351 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:06.395 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:06.461 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:06.594 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:06.644 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:06.710 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:06.838 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:06.864 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:06.914 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:07.070 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:07.190 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:07.222 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:07.234 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:07.258 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:07.275 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:07.307 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:07.342 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:07.379 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:07.417 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:07.498 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:07.609 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:07.676 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:07.783 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:07.831 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:07.884 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:07.930 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:07.944 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:07.954 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:08.000 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:08.012 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:08.018 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:08.040 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:08.112 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:08.133 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:08.205 INFO:teuthology.orchestra.run.smithi060.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 59 kB/s | 221 kB 00:03 2024-03-25T17:47:08.230 INFO:teuthology.orchestra.run.smithi060.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 13 kB/s | 36 kB 00:02 2024-03-25T17:47:08.251 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:08.339 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:08.456 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:08.475 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:08.621 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:08.655 INFO:teuthology.orchestra.run.smithi090.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 247 kB/s | 742 kB 00:03 2024-03-25T17:47:08.701 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:08.720 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:08.780 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:08.822 INFO:teuthology.orchestra.run.smithi060.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 522 kB/s | 309 kB 00:00 2024-03-25T17:47:08.937 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:09.293 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:09.771 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:10.111 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:10.178 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:10.309 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:10.345 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:10.409 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:10.536 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:10.645 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:10.730 INFO:teuthology.orchestra.run.smithi090.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 240 kB/s | 498 kB 00:02 2024-03-25T17:47:10.798 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:10.961 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:11.315 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:11.357 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:11.457 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:11.492 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:11.571 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:11.627 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:11.645 INFO:teuthology.orchestra.run.smithi060.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:11.739 INFO:teuthology.orchestra.run.smithi060.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 13 kB/s | 47 kB 00:03 2024-03-25T17:47:11.931 INFO:teuthology.orchestra.run.smithi060.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 135 kB/s | 26 kB 00:00 2024-03-25T17:47:12.115 INFO:teuthology.orchestra.run.smithi060.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 265 kB/s | 49 kB 00:00 2024-03-25T17:47:12.306 INFO:teuthology.orchestra.run.smithi060.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 352 kB/s | 67 kB 00:00 2024-03-25T17:47:13.323 INFO:teuthology.orchestra.run.smithi060.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 824 kB/s | 838 kB 00:01 2024-03-25T17:47:13.591 INFO:teuthology.orchestra.run.smithi060.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 719 kB/s | 192 kB 00:00 2024-03-25T17:47:14.441 INFO:teuthology.orchestra.run.smithi060.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 805 kB/s | 4.4 MB 00:05 2024-03-25T17:47:15.616 INFO:teuthology.orchestra.run.smithi060.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 800 kB/s | 1.6 MB 00:02 2024-03-25T17:47:16.264 INFO:teuthology.orchestra.run.smithi090.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 350 kB/s | 9.7 MB 00:28 2024-03-25T17:47:16.448 INFO:teuthology.orchestra.run.smithi090.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 246 kB/s | 45 kB 00:00 2024-03-25T17:47:17.256 INFO:teuthology.orchestra.run.smithi090.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 162 kB/s | 131 kB 00:00 2024-03-25T17:47:17.998 INFO:teuthology.orchestra.run.smithi090.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 218 kB/s | 162 kB 00:00 2024-03-25T17:47:19.257 INFO:teuthology.orchestra.run.smithi090.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 255 kB/s | 320 kB 00:01 2024-03-25T17:47:20.499 INFO:teuthology.orchestra.run.smithi090.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 240 kB/s | 298 kB 00:01 2024-03-25T17:47:20.908 INFO:teuthology.orchestra.run.smithi090.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 243 kB/s | 99 kB 00:00 2024-03-25T17:47:21.041 INFO:teuthology.orchestra.run.smithi090.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 795 kB/s | 106 kB 00:00 2024-03-25T17:47:21.218 INFO:teuthology.orchestra.run.smithi090.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T17:47:21.218 INFO:teuthology.orchestra.run.smithi090.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T17:47:21.218 INFO:teuthology.orchestra.run.smithi090.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T17:47:21.218 INFO:teuthology.orchestra.run.smithi090.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T17:47:21.260 INFO:teuthology.orchestra.run.smithi090.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2024-03-25T17:47:21.295 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:21.341 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:21.432 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:21.592 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:21.945 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:21.986 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:22.074 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:22.153 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:22.293 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:22.419 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:22.452 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:22.522 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:22.653 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:22.717 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:22.844 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:22.866 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:22.921 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:23.397 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:23.725 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:23.783 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:23.802 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:23.964 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:24.315 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:24.377 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:24.507 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:24.671 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:24.760 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:24.885 INFO:teuthology.orchestra.run.smithi090.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 61 kB/s | 221 kB 00:03 2024-03-25T17:47:24.920 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:24.965 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:25.061 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:25.222 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:25.576 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:25.617 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:25.708 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:25.783 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:25.923 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:26.046 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:26.065 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:26.139 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:26.272 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:26.338 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:26.440 INFO:teuthology.orchestra.run.smithi060.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 288 kB/s | 3.4 MB 00:11 2024-03-25T17:47:26.484 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:26.501 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:26.556 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:26.690 INFO:teuthology.orchestra.run.smithi060.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 289 kB/s | 3.1 MB 00:11 2024-03-25T17:47:27.035 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:27.359 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:27.414 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:27.431 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:27.590 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:27.760 INFO:teuthology.orchestra.run.smithi090.stdout:(19/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 307 kB/s | 5.1 MB 00:17 2024-03-25T17:47:27.823 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T17:47:27.863 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:27.960 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirrors.iu13.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 206.82.17.213) 2024-03-25T17:47:28.032 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:28.067 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:28.120 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:28.147 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:28.310 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:28.398 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:28.494 INFO:teuthology.orchestra.run.smithi090.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 9.9 kB/s | 36 kB 00:03 2024-03-25T17:47:28.513 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T17:47:28.554 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:28.594 INFO:teuthology.orchestra.run.smithi090.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-03-25T17:47:28.665 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.servaxnet.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 162.255.226.75) 2024-03-25T17:47:28.761 INFO:teuthology.orchestra.run.smithi090.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-03-25T17:47:28.795 INFO:teuthology.orchestra.run.smithi090.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 784 kB/s | 26 kB 00:00 2024-03-25T17:47:28.799 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:28.828 INFO:teuthology.orchestra.run.smithi090.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-25T17:47:28.888 INFO:teuthology.orchestra.run.smithi090.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T17:47:28.939 INFO:teuthology.orchestra.run.smithi090.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-03-25T17:47:28.939 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://repos.eggycrew.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.167.241.199) 2024-03-25T17:47:28.981 INFO:teuthology.orchestra.run.smithi090.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-03-25T17:47:29.056 INFO:teuthology.orchestra.run.smithi090.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-25T17:47:29.122 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:29.140 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.pilotfiber.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.158.133.254) 2024-03-25T17:47:29.225 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:29.366 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T17:47:29.431 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:29.555 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T17:47:29.575 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:29.629 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:30.124 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:30.448 INFO:teuthology.orchestra.run.smithi060.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 335 kB/s | 20 MB 01:00 2024-03-25T17:47:30.457 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:47:30.458 INFO:teuthology.orchestra.run.smithi060.stdout:Total 969 kB/s | 58 MB 01:01 2024-03-25T17:47:30.466 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://centos-stream-distro.1gservers.com/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 104.251.122.5) 2024-03-25T17:47:30.489 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:47:30.522 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:30.531 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:47:30.531 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:47:30.539 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T17:47:30.702 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:31.022 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:47:31.022 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:47:31.056 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.facebook.net/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.114.132.32) 2024-03-25T17:47:31.093 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:31.170 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://atl.mirrors.knownhost.com/centos/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 67.222.0.49) 2024-03-25T17:47:31.312 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:31.401 INFO:teuthology.orchestra.run.smithi090.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T17:47:31.498 INFO:teuthology.orchestra.run.smithi090.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 13 kB/s | 47 kB 00:03 2024-03-25T17:47:32.645 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:47:32.731 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:47:32.761 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:47:32.786 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:47:32.809 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:32.826 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:32.960 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:33.023 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:33.385 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:47:33.580 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:33.673 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:33.762 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:47:33.798 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:33.844 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:33.873 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:47:33.917 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:47:33.987 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:47:34.009 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:47:34.074 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:47:34.107 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T17:47:34.218 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:47:34.250 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:47:34.291 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:47:34.324 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:47:34.353 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:47:34.394 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:47:34.731 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:47:34.757 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:47:34.864 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:47:34.895 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:47:34.942 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:47:35.431 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:47:35.460 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:47:35.531 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:47:35.601 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:47:35.637 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:47:37.924 INFO:teuthology.orchestra.run.smithi090.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 499 kB/s | 3.1 MB 00:06 2024-03-25T17:47:37.925 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:37.958 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:38.530 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:47:38.570 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:47:38.656 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:47:38.656 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:47:38.656 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:47:38.656 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:47:38.711 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:40.298 INFO:teuthology.orchestra.run.smithi090.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 308 kB/s | 3.4 MB 00:11 2024-03-25T17:47:48.352 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-03-25T17:47:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-03-25T17:47:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-03-25T17:47:48.352 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-03-25T17:47:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-03-25T17:47:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-03-25T17:47:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-03-25T17:47:48.353 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:47:49.290 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:47:49.313 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:47:49.313 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:47:49.314 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:47:49.314 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:47:49.314 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:47:49.314 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:47:49.315 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:47:49.393 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:47:49.393 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:50.538 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:50.538 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:47:50.538 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:47:50.538 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:47:50.538 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:47:50.539 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:47:50.540 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:47:50.541 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:47:50.541 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:47:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:47:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:47:51.600 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-03-25T17:47:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.600 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:47:51.601 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:51.602 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T17:47:51.603 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T17:47:51.603 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T17:47:51.603 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:47:51.603 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:47:51.603 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:47:51.603 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:47:51.953 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test 2024-03-25T17:47:52.524 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:26 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:47:52.688 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:47:52.688 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:47:52.688 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:47:52.688 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-03-25T17:47:52.689 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:47:52.690 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 48 M 2024-03-25T17:47:52.690 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 204 M 2024-03-25T17:47:52.690 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:47:52.897 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T17:47:52.947 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T17:47:52.981 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T17:47:53.198 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 863 kB/s | 187 kB 00:00 2024-03-25T17:47:53.490 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 214 kB/s | 163 kB 00:00 2024-03-25T17:47:56.682 INFO:teuthology.orchestra.run.smithi090.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 295 kB/s | 20 MB 01:08 2024-03-25T17:47:56.690 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:47:56.690 INFO:teuthology.orchestra.run.smithi090.stdout:Total 858 kB/s | 58 MB 01:09 2024-03-25T17:47:56.720 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:47:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:47:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:47:57.238 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:47:57.238 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:47:58.676 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:47:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:47:58.795 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:47:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:47:58.852 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:58.869 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T17:47:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:59.082 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T17:47:59.409 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:47:59.499 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:47:59.696 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:47:59.726 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:47:59.831 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:47:59.880 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:47:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:47:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:48:00.060 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:48:00.087 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T17:48:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:48:00.221 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:48:00.262 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:48:00.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:48:00.315 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:48:00.348 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:48:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:48:00.704 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:48:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:48:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:48:00.866 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:48:01.316 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:48:01.342 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:48:01.396 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:48:01.463 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:48:01.494 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:48:03.683 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:48:03.734 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:48:04.318 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:48:04.373 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:48:04.465 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:48:04.465 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:48:04.465 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:48:04.465 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:48:04.499 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:48:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-03-25T17:48:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:48:15.419 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:48:15.443 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:48:15.443 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:48:15.443 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:48:15.443 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:48:15.444 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:48:15.444 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:48:15.446 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:48:15.539 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:48:15.540 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:48:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:48:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:48:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:48:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:48:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:48:16.882 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:48:16.883 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:48:16.884 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:48:17.995 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:48:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:48:17.997 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:48:18.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-03-25T17:48:18.949 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:33 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:48:19.115 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:48:19.116 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:48:19.116 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:48:19.116 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:48:19.116 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:48:19.116 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-03-25T17:48:19.117 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:48:19.118 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 48 M 2024-03-25T17:48:19.118 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 204 M 2024-03-25T17:48:19.118 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:48:19.373 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T17:48:19.440 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.9 MB/s | 300 kB 00:00 2024-03-25T17:48:19.474 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T17:48:19.974 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 374 kB/s | 187 kB 00:00 2024-03-25T17:48:20.058 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 197 kB/s | 163 kB 00:00 2024-03-25T17:49:34.663 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 475 kB/s | 47 MB 01:41 2024-03-25T17:49:34.665 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:34.665 INFO:teuthology.orchestra.run.smithi060.stdout:Total 484 kB/s | 48 MB 01:41 2024-03-25T17:49:34.676 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:49:34.697 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:49:34.697 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:49:35.089 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:49:35.089 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:49:35.682 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:49:35.764 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:49:35.825 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:49:35.856 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:49:35.906 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:49:35.932 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:49:40.665 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:49:40.742 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:49:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:49:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:49:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:49:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:49:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:49:42.970 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:49:43.380 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:49:43.380 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:49:43.380 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:49:43.381 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:49:43.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph 2024-03-25T17:49:44.285 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:18 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:49:44.453 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:49:44.455 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-03-25T17:49:44.455 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:49:44.455 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-03-25T17:49:44.455 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:49:44.456 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:49:44.457 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:49:44.458 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:49:44.459 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:49:44.459 INFO:teuthology.orchestra.run.smithi060.stdout:========================================================================================= 2024-03-25T17:49:44.459 INFO:teuthology.orchestra.run.smithi060.stdout:Install 36 Packages 2024-03-25T17:49:44.459 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:49:44.459 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 30 M 2024-03-25T17:49:44.459 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 107 M 2024-03-25T17:49:44.460 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:49:45.382 INFO:teuthology.orchestra.run.smithi060.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T17:49:49.757 INFO:teuthology.orchestra.run.smithi060.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 321 kB/s | 1.5 MB 00:04 2024-03-25T17:49:55.882 INFO:teuthology.orchestra.run.smithi060.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 200 kB/s | 2.1 MB 00:10 2024-03-25T17:49:57.148 INFO:teuthology.orchestra.run.smithi060.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 195 kB/s | 247 kB 00:01 2024-03-25T17:49:57.274 INFO:teuthology.orchestra.run.smithi060.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T17:49:57.315 INFO:teuthology.orchestra.run.smithi060.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T17:49:57.349 INFO:teuthology.orchestra.run.smithi060.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-25T17:49:57.383 INFO:teuthology.orchestra.run.smithi060.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T17:49:57.417 INFO:teuthology.orchestra.run.smithi060.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T17:49:57.467 INFO:teuthology.orchestra.run.smithi060.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T17:49:57.542 INFO:teuthology.orchestra.run.smithi060.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T17:49:57.584 INFO:teuthology.orchestra.run.smithi060.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T17:49:57.617 INFO:teuthology.orchestra.run.smithi060.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T17:49:57.659 INFO:teuthology.orchestra.run.smithi060.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T17:49:57.926 INFO:teuthology.orchestra.run.smithi060.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 468 kB/s | 125 kB 00:00 2024-03-25T17:49:58.152 INFO:teuthology.orchestra.run.smithi060.stdout:(16/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 376 kB/s | 4.7 MB 00:12 2024-03-25T17:49:58.277 INFO:teuthology.orchestra.run.smithi060.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 64 kB/s | 22 kB 00:00 2024-03-25T17:49:58.377 INFO:teuthology.orchestra.run.smithi060.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 671 kB/s | 151 kB 00:00 2024-03-25T17:49:58.403 INFO:teuthology.orchestra.run.smithi060.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 346 kB/s | 43 kB 00:00 2024-03-25T17:49:58.453 INFO:teuthology.orchestra.run.smithi060.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-03-25T17:49:58.503 INFO:teuthology.orchestra.run.smithi060.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T17:49:58.528 INFO:teuthology.orchestra.run.smithi060.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-25T17:49:58.579 INFO:teuthology.orchestra.run.smithi060.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T17:49:58.604 INFO:teuthology.orchestra.run.smithi060.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-03-25T17:49:58.646 INFO:teuthology.orchestra.run.smithi060.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-25T17:49:58.671 INFO:teuthology.orchestra.run.smithi060.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-25T17:49:58.738 INFO:teuthology.orchestra.run.smithi060.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 505 kB/s | 46 kB 00:00 2024-03-25T17:49:58.788 INFO:teuthology.orchestra.run.smithi060.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 674 kB/s | 79 kB 00:00 2024-03-25T17:49:58.822 INFO:teuthology.orchestra.run.smithi060.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-25T17:49:58.847 INFO:teuthology.orchestra.run.smithi060.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 283 kB/s | 16 kB 00:00 2024-03-25T17:49:58.898 INFO:teuthology.orchestra.run.smithi060.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T17:49:58.931 INFO:teuthology.orchestra.run.smithi060.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 428 kB/s | 36 kB 00:00 2024-03-25T17:49:58.989 INFO:teuthology.orchestra.run.smithi060.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-25T17:49:59.023 INFO:teuthology.orchestra.run.smithi060.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-25T17:49:59.040 INFO:teuthology.orchestra.run.smithi060.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-03-25T17:50:31.718 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 365 kB/s | 47 MB 02:12 2024-03-25T17:50:31.720 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:31.720 INFO:teuthology.orchestra.run.smithi090.stdout:Total 372 kB/s | 48 MB 02:12 2024-03-25T17:50:31.732 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:50:31.752 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:50:31.752 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:50:32.133 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:50:32.133 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:50:32.726 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:50:32.809 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:50:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:50:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:50:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:50:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:50:36.312 INFO:teuthology.orchestra.run.smithi060.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 369 kB/s | 17 MB 00:46 2024-03-25T17:50:36.317 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:36.317 INFO:teuthology.orchestra.run.smithi060.stdout:Total 586 kB/s | 30 MB 00:51 2024-03-25T17:50:36.341 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:50:36.385 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:50:36.385 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:50:36.805 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:50:36.805 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:50:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:50:37.540 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:50:38.264 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:50:38.331 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:50:38.368 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:50:38.400 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:50:38.475 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:50:38.517 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:50:38.549 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:50:38.638 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:50:38.722 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:50:38.769 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:50:38.968 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:50:39.020 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:50:39.057 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:50:39.124 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:50:39.188 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:50:39.384 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:50:39.437 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:50:39.475 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:50:39.510 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:50:39.539 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:50:39.620 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:50:39.658 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:50:39.691 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:50:39.732 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:50:39.775 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:50:39.810 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:50:39.897 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:50:39.932 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:50:40.009 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:50:40.159 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:50:40.350 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:50:40.437 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:50:40.437 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:50:40.438 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:50:40.438 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:50:40.438 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:50:40.438 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:50:40.524 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:50:40.546 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:50:40.572 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:50:40.572 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:40.572 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:50:40.572 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:50:40.572 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:50:40.572 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:50:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:50:40.840 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:50:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:50:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:50:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:50:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:50:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:50:40.841 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:50:41.246 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-03-25T17:50:41.812 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:03:56 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:50:41.972 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:50:41.974 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:50:41.975 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:50:41.976 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:50:41.977 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout:========================================================================================= 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout:Install 36 Packages 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 30 M 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 107 M 2024-03-25T17:50:41.978 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:50:42.105 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:50:42.131 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:50:42.131 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:42.131 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:50:42.131 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:50:42.131 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:50:42.131 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:42.597 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:50:42.620 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:50:42.620 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:42.620 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:50:42.620 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:50:42.620 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:50:42.621 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:42.846 INFO:teuthology.orchestra.run.smithi090.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T17:50:42.847 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:50:42.871 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:50:42.871 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:42.871 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:50:42.871 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:50:42.871 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:50:42.871 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:42.941 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:50:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:50:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:50:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:50:44.495 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:50:44.496 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:50:44.497 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:50:45.238 INFO:teuthology.orchestra.run.smithi090.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 549 kB/s | 1.5 MB 00:02 2024-03-25T17:50:45.492 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:50:45.492 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:45.492 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:50:45.492 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:45.492 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:45.492 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:50:45.493 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:50:45.494 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:50:45.495 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:50:45.495 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:50:45.495 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:50:45.495 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:50:45.495 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:50:45.495 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:45.495 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:50:45.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base 2024-03-25T17:50:46.425 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:04:20 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:50:46.525 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:50:46.580 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:50:46.581 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:50:46.581 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:50:46.641 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install cephadm 2024-03-25T17:50:47.210 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:50:47.364 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:50:47.365 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 747 k 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 754 k 2024-03-25T17:50:47.366 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:50:47.830 INFO:teuthology.orchestra.run.smithi090.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 405 kB/s | 2.1 MB 00:05 2024-03-25T17:50:48.639 INFO:teuthology.orchestra.run.smithi090.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 306 kB/s | 247 kB 00:00 2024-03-25T17:50:48.756 INFO:teuthology.orchestra.run.smithi090.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T17:50:48.798 INFO:teuthology.orchestra.run.smithi090.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T17:50:48.832 INFO:teuthology.orchestra.run.smithi090.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-25T17:50:48.865 INFO:teuthology.orchestra.run.smithi090.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T17:50:48.899 INFO:teuthology.orchestra.run.smithi090.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T17:50:48.949 INFO:teuthology.orchestra.run.smithi090.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T17:50:49.034 INFO:teuthology.orchestra.run.smithi090.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T17:50:49.068 INFO:teuthology.orchestra.run.smithi090.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T17:50:49.110 INFO:teuthology.orchestra.run.smithi090.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T17:50:49.152 INFO:teuthology.orchestra.run.smithi090.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T17:50:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 881 kB/s | 125 kB 00:00 2024-03-25T17:50:49.894 INFO:teuthology.orchestra.run.smithi090.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-03-25T17:50:49.939 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 291 kB/s | 747 kB 00:02 2024-03-25T17:50:49.939 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:49.940 INFO:teuthology.orchestra.run.smithi060.stdout:Total 290 kB/s | 747 kB 00:02 2024-03-25T17:50:49.940 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:50:49.947 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:50:49.947 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:50:49.953 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:50:49.953 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:50:50.030 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:50:50.153 INFO:teuthology.orchestra.run.smithi090.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 586 kB/s | 151 kB 00:00 2024-03-25T17:50:50.245 INFO:teuthology.orchestra.run.smithi090.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 473 kB/s | 43 kB 00:00 2024-03-25T17:50:50.404 INFO:teuthology.orchestra.run.smithi090.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-25T17:50:50.510 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:50.529 INFO:teuthology.orchestra.run.smithi090.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-25T17:50:50.579 INFO:teuthology.orchestra.run.smithi090.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-25T17:50:50.605 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:50.629 INFO:teuthology.orchestra.run.smithi090.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-25T17:50:50.680 INFO:teuthology.orchestra.run.smithi090.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-25T17:50:50.730 INFO:teuthology.orchestra.run.smithi090.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-25T17:50:50.781 INFO:teuthology.orchestra.run.smithi090.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-03-25T17:50:50.848 INFO:teuthology.orchestra.run.smithi090.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 694 kB/s | 46 kB 00:00 2024-03-25T17:50:50.906 INFO:teuthology.orchestra.run.smithi090.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T17:50:50.976 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:50.982 INFO:teuthology.orchestra.run.smithi090.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T17:50:51.032 INFO:teuthology.orchestra.run.smithi090.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-03-25T17:50:51.091 INFO:teuthology.orchestra.run.smithi090.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T17:50:51.141 INFO:teuthology.orchestra.run.smithi090.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-03-25T17:50:51.208 INFO:teuthology.orchestra.run.smithi090.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T17:50:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:50:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:50:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:50:51.279 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:51.279 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:50:51.302 INFO:teuthology.orchestra.run.smithi090.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-25T17:50:51.360 INFO:teuthology.orchestra.run.smithi090.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-25T17:50:51.503 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:50:52.084 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:04:26 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:50:52.238 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:50:52.239 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:50:52.239 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 145 k 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 436 k 2024-03-25T17:50:52.240 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:50:53.037 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 182 kB/s | 145 kB 00:00 2024-03-25T17:50:53.038 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:53.038 INFO:teuthology.orchestra.run.smithi060.stdout:Total 182 kB/s | 145 kB 00:00 2024-03-25T17:50:53.038 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:50:53.048 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:50:53.048 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:50:53.107 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:50:53.107 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:50:53.264 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:50:53.363 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:50:53.388 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:50:53.388 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:50:53.388 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:50:53.388 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:54.398 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:50:54.398 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:54.398 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:50:54.398 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:54.398 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:54.398 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:50:54.684 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr 2024-03-25T17:50:55.257 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:04:29 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:50:55.358 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:50:55.413 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:50:55.414 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:50:55.414 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:50:55.485 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:50:55.719 INFO:teuthology.orchestra.run.smithi090.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 373 kB/s | 4.7 MB 00:12 2024-03-25T17:50:56.053 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:04:30 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:50:56.208 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:50:56.210 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout:====================================================================================== 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:50:56.211 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 6.9 M 2024-03-25T17:50:56.212 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-03-25T17:50:56.212 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:50:56.907 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 30 kB/s | 15 kB 00:00 2024-03-25T17:50:56.933 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 59 kB/s | 31 kB 00:00 2024-03-25T17:50:57.075 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T17:50:57.108 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-25T17:50:57.359 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-25T17:50:57.459 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T17:50:57.559 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 97 kB/s | 19 kB 00:00 2024-03-25T17:50:57.668 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.8 MB/s | 2.0 MB 00:00 2024-03-25T17:50:57.693 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T17:50:57.718 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 611 kB/s | 31 kB 00:00 2024-03-25T17:50:57.785 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-25T17:51:02.260 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 311 kB/s | 1.8 MB 00:05 2024-03-25T17:51:02.266 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:02.266 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T17:51:02.286 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:51:02.305 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:51:02.305 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:51:02.483 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:51:02.483 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:51:03.125 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:51:03.194 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:51:03.230 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:51:03.277 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:51:03.323 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:51:03.423 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:51:03.554 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:51:03.655 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:51:03.712 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:51:03.739 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:51:03.763 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:51:04.009 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:51:04.046 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:51:04.842 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:51:04.843 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:51:05.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:51:05.206 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:05.206 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:51:05.206 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:51:05.206 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:05.206 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:05.207 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:51:05.457 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:51:06.026 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:51:06.183 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:51:06.184 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:51:06.185 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================================ 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-03-25T17:51:06.186 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:06.187 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 37 M 2024-03-25T17:51:06.187 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 209 M 2024-03-25T17:51:06.187 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:51:06.375 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 289 kB/s | 32 kB 00:00 2024-03-25T17:51:06.434 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 503 kB/s | 17 kB 00:00 2024-03-25T17:51:06.467 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T17:51:06.693 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-25T17:51:06.818 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 39 MB/s | 4.8 MB 00:00 2024-03-25T17:51:06.860 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T17:51:07.060 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-03-25T17:51:07.294 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-25T17:51:07.511 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-25T17:51:07.536 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 809 kB/s | 195 kB 00:00 2024-03-25T17:51:07.652 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-25T17:51:27.894 INFO:teuthology.orchestra.run.smithi090.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 403 kB/s | 17 MB 00:42 2024-03-25T17:51:27.897 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:27.898 INFO:teuthology.orchestra.run.smithi090.stdout:Total 661 kB/s | 30 MB 00:45 2024-03-25T17:51:27.920 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:51:27.962 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:51:27.962 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:51:28.391 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:51:28.392 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:51:29.740 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:51:29.813 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:51:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:51:29.899 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:51:29.969 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:51:30.015 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:51:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:51:30.147 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:51:30.244 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:51:30.293 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:51:30.365 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:51:30.420 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:51:30.458 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:51:30.515 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:51:30.571 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:51:30.755 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:51:30.806 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:51:30.845 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:51:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:51:30.910 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:51:30.976 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:51:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:51:31.046 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:51:31.086 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:51:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:51:31.164 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:51:31.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:51:31.295 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:51:31.375 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:51:31.511 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:51:31.632 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:51:31.790 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:51:31.819 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:51:31.844 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:51:31.844 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:31.844 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:51:31.844 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:51:31.844 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:51:31.844 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:33.380 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:51:33.406 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:51:33.406 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:33.407 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:51:33.407 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:51:33.407 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:51:33.407 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:33.847 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:51:33.870 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:51:33.870 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:33.870 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:51:33.870 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:51:33.870 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:51:33.870 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:34.081 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:51:34.104 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:51:34.104 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:34.104 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:51:34.105 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:51:34.105 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:51:34.105 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:34.171 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:51:36.179 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:51:36.180 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:51:36.181 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:51:36.350 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 252 kB/s | 7.4 MB 00:30 2024-03-25T17:51:36.352 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:36.353 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.2 MB/s | 37 MB 00:30 2024-03-25T17:51:36.394 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:51:36.421 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:51:36.421 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:51:36.797 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:51:36.798 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:51:37.173 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:51:37.173 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:37.173 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:51:37.173 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:51:37.174 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:51:37.175 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:37.176 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:51:37.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-03-25T17:51:37.666 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:51:37.748 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:51:37.779 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:51:37.878 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:51:38.042 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:51:38.147 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:51:38.226 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:51:38.243 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:38.265 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:51:38.295 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:51:38.296 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:51:38.296 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:51:38.346 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:51:38.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-03-25T17:51:38.933 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:51:39.024 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:51:39.082 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:51:39.082 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:51:39.082 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 747 k 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 754 k 2024-03-25T17:51:39.083 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:51:39.141 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:51:40.453 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:51:40.872 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 418 kB/s | 747 kB 00:01 2024-03-25T17:51:40.873 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:40.873 INFO:teuthology.orchestra.run.smithi090.stdout:Total 417 kB/s | 747 kB 00:01 2024-03-25T17:51:40.874 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:51:40.879 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:51:40.880 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:51:40.886 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:51:40.886 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:51:40.965 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:51:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:41.538 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:51:41.564 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:51:41.967 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:42.265 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:51:42.265 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:42.265 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:51:42.265 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:42.266 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:42.266 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:51:42.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:51:43.091 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:51:43.246 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 145 k 2024-03-25T17:51:43.247 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 436 k 2024-03-25T17:51:43.248 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:51:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:51:44.075 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:51:44.076 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:51:44.126 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 165 kB/s | 145 kB 00:00 2024-03-25T17:51:44.127 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:44.127 INFO:teuthology.orchestra.run.smithi090.stdout:Total 165 kB/s | 145 kB 00:00 2024-03-25T17:51:44.128 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:51:44.137 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:51:44.137 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:51:44.193 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:51:44.193 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:51:44.332 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:51:44.410 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:51:44.434 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:51:44.435 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:51:44.435 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:51:44.435 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:51:44.484 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:51:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:51:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:51:44.485 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:44.485 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:51:44.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:51:45.459 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:51:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:51:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:51:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:45.461 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:45.461 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:51:45.615 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:51:45.616 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T17:51:45.616 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:51:45.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout:Install 13 Packages 2024-03-25T17:51:45.618 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:45.619 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.0 M 2024-03-25T17:51:45.619 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-03-25T17:51:45.619 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:51:45.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-03-25T17:51:45.906 INFO:teuthology.orchestra.run.smithi060.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 240 kB/s | 24 kB 00:00 2024-03-25T17:51:45.956 INFO:teuthology.orchestra.run.smithi060.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-25T17:51:45.998 INFO:teuthology.orchestra.run.smithi060.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T17:51:46.023 INFO:teuthology.orchestra.run.smithi060.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T17:51:46.048 INFO:teuthology.orchestra.run.smithi060.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T17:51:46.073 INFO:teuthology.orchestra.run.smithi060.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 873 kB/s | 43 kB 00:00 2024-03-25T17:51:46.266 INFO:teuthology.orchestra.run.smithi060.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2024-03-25T17:51:46.290 INFO:teuthology.orchestra.run.smithi060.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-03-25T17:51:46.353 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:51:46.358 INFO:teuthology.orchestra.run.smithi060.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 94 kB/s | 52 kB 00:00 2024-03-25T17:51:46.391 INFO:teuthology.orchestra.run.smithi060.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.6 MB/s | 203 kB 00:00 2024-03-25T17:51:46.453 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:51:46.483 INFO:teuthology.orchestra.run.smithi060.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T17:51:46.506 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:51:46.507 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:51:46.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:51:46.608 INFO:teuthology.orchestra.run.smithi060.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 236 kB/s | 59 kB 00:00 2024-03-25T17:51:46.859 INFO:teuthology.orchestra.run.smithi060.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-25T17:51:46.863 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:46.863 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T17:51:46.863 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:51:46.878 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:51:46.878 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:51:46.975 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:51:46.976 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:51:47.133 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:51:47.283 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:51:47.284 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-03-25T17:51:47.285 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:47.286 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 6.9 M 2024-03-25T17:51:47.286 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-03-25T17:51:47.286 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:51:47.497 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:51:47.664 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:51:47.720 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:51:47.784 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:51:47.820 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:51:47.862 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:51:47.944 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 31 kB/s | 15 kB 00:00 2024-03-25T17:51:47.967 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:51:47.969 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 60 kB/s | 31 kB 00:00 2024-03-25T17:51:48.077 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:51:48.130 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:51:48.228 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 920 kB/s | 238 kB 00:00 2024-03-25T17:51:48.336 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-03-25T17:51:48.645 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T17:51:48.704 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-03-25T17:51:48.846 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.4 MB/s | 863 kB 00:00 2024-03-25T17:51:48.894 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:51:48.896 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-25T17:51:48.951 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:51:48.963 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 460 kB/s | 31 kB 00:00 2024-03-25T17:51:49.004 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:51:49.022 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-25T17:51:49.061 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:51:49.072 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 639 kB/s | 144 kB 00:00 2024-03-25T17:51:49.089 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:51:49.903 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:51:49.903 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:51:49.904 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:51:49.905 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:51:49.905 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:51:49.905 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:51:50.276 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:51:50.277 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:51:50.277 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:51:50.277 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:51:50.277 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:51:50.277 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:50.277 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:51:50.528 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:51:51.104 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:51:51.261 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:51:51.262 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T17:51:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:51:51.262 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T17:51:51.262 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:51:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:51:51.262 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:51:51.262 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-03-25T17:51:51.263 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:51.264 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 6.8 M 2024-03-25T17:51:51.264 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 33 M 2024-03-25T17:51:51.264 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:51:52.187 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-25T17:51:52.220 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 639 kB/s | 248 kB 00:00 2024-03-25T17:51:52.463 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 219 kB/s | 138 kB 00:00 2024-03-25T17:51:52.830 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 95 kB/s | 58 kB 00:00 2024-03-25T17:51:53.047 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 326 kB/s | 1.8 MB 00:05 2024-03-25T17:51:53.051 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:53.052 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.2 MB/s | 6.9 MB 00:05 2024-03-25T17:51:53.072 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:51:53.091 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:51:53.092 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:51:53.196 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 69 kB/s | 51 kB 00:00 2024-03-25T17:51:53.264 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:51:53.265 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:51:53.794 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:51:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:51:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:51:53.972 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:51:54.012 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:51:54.124 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:51:54.254 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:51:54.272 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 263 kB/s | 548 kB 00:02 2024-03-25T17:51:54.275 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:54.276 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.3 MB/s | 6.8 MB 00:03 2024-03-25T17:51:54.289 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:51:54.300 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:51:54.300 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:51:54.349 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:51:54.394 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:51:54.394 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:51:54.395 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:51:54.423 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:51:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:51:54.699 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:51:54.703 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:51:54.738 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:51:54.873 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T17:51:54.916 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T17:51:55.465 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T17:51:55.554 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T17:51:55.622 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T17:51:55.652 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:51:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:51:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:51:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:51:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:51:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:51:55.672 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:51:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:51:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:51:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:51:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:51:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:51:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:51:56.002 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:51:56.002 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:56.002 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:51:56.002 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:51:56.002 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:56.003 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:51:56.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:51:56.436 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:51:56.436 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T17:51:56.436 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T17:51:56.436 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T17:51:56.436 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T17:51:56.436 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T17:51:56.741 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T17:51:56.741 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:56.741 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:51:56.741 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:51:56.741 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T17:51:56.742 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T17:51:56.742 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T17:51:56.742 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T17:51:56.742 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T17:51:56.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:56.742 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:51:56.865 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:51:56.973 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse 2024-03-25T17:51:57.015 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:51:57.016 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:51:57.017 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================================ 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:51:57.018 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 37 M 2024-03-25T17:51:57.019 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 209 M 2024-03-25T17:51:57.019 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:51:57.547 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:51:57.578 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 154 kB/s | 32 kB 00:00 2024-03-25T17:51:57.628 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 345 kB/s | 17 kB 00:00 2024-03-25T17:51:57.695 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 518 kB/s | 34 kB 00:00 2024-03-25T17:51:57.703 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:51:57.703 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:51:57.703 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-03-25T17:51:57.704 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:51:57.705 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 964 k 2024-03-25T17:51:57.705 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.8 M 2024-03-25T17:51:57.705 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:51:57.915 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 375 kB/s | 79 kB 00:00 2024-03-25T17:51:58.029 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-25T17:51:58.246 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.4 MB/s | 3.0 MB 00:00 2024-03-25T17:51:58.321 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-25T17:51:58.346 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-25T17:51:58.522 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T17:51:58.572 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-25T17:51:58.831 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-03-25T17:51:58.964 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 629 kB/s | 245 kB 00:00 2024-03-25T17:52:00.673 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 298 kB/s | 885 kB 00:02 2024-03-25T17:52:00.674 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:00.675 INFO:teuthology.orchestra.run.smithi060.stdout:Total 324 kB/s | 964 kB 00:02 2024-03-25T17:52:00.675 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:52:00.700 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:52:00.700 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:52:00.781 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:52:00.781 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:52:00.979 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:52:01.152 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T17:52:01.217 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:52:01.873 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:52:01.873 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:52:02.166 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T17:52:02.166 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:02.166 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:52:02.166 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T17:52:02.166 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:02.166 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:02.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-volume 2024-03-25T17:52:03.022 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:37 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:03.181 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-03-25T17:52:03.183 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:03.183 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 332 k 2024-03-25T17:52:03.183 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.5 M 2024-03-25T17:52:03.183 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:52:03.294 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 616 kB/s | 69 kB 00:00 2024-03-25T17:52:04.144 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 274 kB/s | 264 kB 00:00 2024-03-25T17:52:04.146 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:04.146 INFO:teuthology.orchestra.run.smithi060.stdout:Total 345 kB/s | 332 kB 00:00 2024-03-25T17:52:04.146 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:52:04.157 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:52:04.157 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:52:04.200 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:52:04.201 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:52:04.361 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:52:04.543 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T17:52:04.580 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:52:04.596 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:52:04.596 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:52:04.596 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T17:52:04.596 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:05.301 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T17:52:05.622 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T17:52:05.623 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:05.623 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:52:05.623 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:05.623 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T17:52:05.623 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:05.623 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:05.954 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel 2024-03-25T17:52:06.533 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:06.690 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:06.691 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:06.691 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T17:52:06.691 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:06.691 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 127 k 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 456 k 2024-03-25T17:52:06.692 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:52:07.450 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 167 kB/s | 127 kB 00:00 2024-03-25T17:52:07.451 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:07.451 INFO:teuthology.orchestra.run.smithi060.stdout:Total 167 kB/s | 127 kB 00:00 2024-03-25T17:52:07.451 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:52:07.458 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:52:07.458 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:52:07.521 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:52:07.521 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:52:07.670 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:52:07.760 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:08.202 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:08.495 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:08.495 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:08.495 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:52:08.495 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:08.495 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:08.495 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:08.761 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2 2024-03-25T17:52:09.337 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:09.437 INFO:teuthology.orchestra.run.smithi060.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:09.493 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:09.494 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:52:09.494 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:09.563 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs-devel 2024-03-25T17:52:10.137 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:44 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:10.292 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:10.293 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:52:10.294 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:10.294 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 31 k 2024-03-25T17:52:10.294 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 140 k 2024-03-25T17:52:10.294 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:52:11.111 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 38 kB/s | 31 kB 00:00 2024-03-25T17:52:11.112 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:11.112 INFO:teuthology.orchestra.run.smithi060.stdout:Total 38 kB/s | 31 kB 00:00 2024-03-25T17:52:11.112 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:52:11.116 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:52:11.116 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:52:11.146 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:52:11.147 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:52:11.276 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:52:11.372 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:11.772 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:12.037 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:12.037 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:12.037 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:52:12.038 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:12.038 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:12.038 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:12.212 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2 2024-03-25T17:52:12.786 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:12.886 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:12.940 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:12.942 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:52:12.942 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:12.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1 2024-03-25T17:52:13.584 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:47 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:13.687 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:13.743 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:13.745 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:52:13.745 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:13.801 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados 2024-03-25T17:52:14.380 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:48 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:14.480 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:14.534 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:14.536 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:52:14.536 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:14.585 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw 2024-03-25T17:52:15.155 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:15.255 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:15.312 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:15.313 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:52:15.313 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:15.365 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs 2024-03-25T17:52:15.936 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:16.036 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:16.093 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:16.094 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:52:16.094 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:16.146 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd 2024-03-25T17:52:16.717 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:50 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:16.817 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:16.870 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:16.871 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T17:52:16.871 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:16.920 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse 2024-03-25T17:52:17.492 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:51 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:17.648 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 85 k 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 231 k 2024-03-25T17:52:17.649 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:52:18.315 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 128 kB/s | 85 kB 00:00 2024-03-25T17:52:18.315 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:18.316 INFO:teuthology.orchestra.run.smithi060.stdout:Total 128 kB/s | 85 kB 00:00 2024-03-25T17:52:18.316 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:52:18.324 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:52:18.324 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:52:18.383 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:52:18.383 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:52:18.528 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:52:18.642 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:19.016 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:19.309 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:52:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:19.309 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:19.309 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:19.559 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2024-03-25T17:52:20.132 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:20.287 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:20.288 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:20.288 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T17:52:20.288 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:20.288 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 3.1 M 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 13 M 2024-03-25T17:52:20.289 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:52:22.331 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 303 kB/s | 7.4 MB 00:24 2024-03-25T17:52:22.335 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:22.335 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.5 MB/s | 37 MB 00:25 2024-03-25T17:52:22.378 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:52:22.404 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:52:22.404 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:52:22.763 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:52:22.763 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:52:23.474 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:52:23.554 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:52:23.594 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:52:23.695 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:52:23.740 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:52:23.908 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:52:23.939 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:52:24.031 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:52:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:52:24.804 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:52:26.034 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:52:27.058 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:52:27.080 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:52:30.249 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:52:30.250 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:52:30.250 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:52:30.250 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:52:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:52:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:30.652 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:52:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:52:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:30.653 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:31.143 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 291 kB/s | 3.1 MB 00:10 2024-03-25T17:52:31.143 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:31.144 INFO:teuthology.orchestra.run.smithi060.stdout:Total 291 kB/s | 3.1 MB 00:10 2024-03-25T17:52:31.144 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:52:31.156 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:52:31.156 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:52:31.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:52:31.236 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:52:31.236 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:52:31.688 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:52:31.768 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:31.781 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:31.792 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:31.792 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:52:31.793 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T17:52:31.793 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:52:31.793 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:52:31.793 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:31.932 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:31.933 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T17:52:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:52:31.933 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T17:52:31.933 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:52:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:52:31.933 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:52:31.933 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:31.934 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:52:31.935 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T17:52:31.935 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-03-25T17:52:31.935 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:31.935 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-03-25T17:52:31.935 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-03-25T17:52:31.935 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:52:32.188 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 248 kB/s | 24 kB 00:00 2024-03-25T17:52:32.238 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-03-25T17:52:32.297 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-25T17:52:32.330 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-25T17:52:32.364 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-25T17:52:32.390 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 732 kB/s | 43 kB 00:00 2024-03-25T17:52:32.465 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-03-25T17:52:32.491 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 148 kB/s | 15 kB 00:00 2024-03-25T17:52:32.575 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.8 MB/s | 203 kB 00:00 2024-03-25T17:52:32.609 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-03-25T17:52:32.651 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-03-25T17:52:32.726 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 82 kB/s | 52 kB 00:00 2024-03-25T17:52:32.760 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-25T17:52:32.761 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:32.762 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-25T17:52:32.762 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:52:32.774 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:52:32.774 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:52:32.864 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:52:32.864 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:52:32.910 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:32.911 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:32.911 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:52:32.911 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:32.911 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:32.911 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:33.189 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd 2024-03-25T17:52:33.373 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:52:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:52:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:52:33.676 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:52:33.713 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:52:33.763 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:52:33.768 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:33.866 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:52:33.924 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:33.924 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:33.924 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-03-25T17:52:33.924 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 171 k 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 494 k 2024-03-25T17:52:33.925 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:52:33.976 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:52:34.033 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:52:34.762 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:52:34.823 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:52:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:52:34.888 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 178 kB/s | 171 kB 00:00 2024-03-25T17:52:34.888 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:34.889 INFO:teuthology.orchestra.run.smithi060.stdout:Total 178 kB/s | 171 kB 00:00 2024-03-25T17:52:34.889 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:52:34.899 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:52:34.899 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:52:34.906 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:52:34.936 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:52:34.963 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:52:34.964 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:52:35.117 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:52:35.223 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:35.779 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:35.910 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:52:35.910 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:52:35.910 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:52:35.910 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:52:35.910 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:52:35.911 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:52:36.134 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:36.134 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:36.134 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:52:36.135 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:36.135 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:36.135 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:52:36.253 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:52:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:52:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:52:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:52:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:52:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:52:36.254 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:36.254 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:36.458 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:52:36.543 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:52:37.037 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:52:37.106 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:37.192 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:52:37.192 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:37.192 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-03-25T17:52:37.193 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:52:37.194 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 91 M 2024-03-25T17:52:37.194 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 498 M 2024-03-25T17:52:37.194 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T17:52:37.258 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:52:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:37.260 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 6.8 M 2024-03-25T17:52:37.261 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2024-03-25T17:52:37.261 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:52:37.545 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T17:52:37.712 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-25T17:52:37.871 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 365 kB/s | 58 kB 00:00 2024-03-25T17:52:37.946 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 678 kB/s | 51 kB 00:00 2024-03-25T17:52:37.980 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T17:52:38.364 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 138 kB/s | 138 kB 00:00 2024-03-25T17:52:38.365 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:38.365 INFO:teuthology.orchestra.run.smithi090.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-03-25T17:52:38.378 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:52:38.388 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:52:38.389 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:52:38.478 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:52:38.479 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:52:38.780 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:52:38.957 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T17:52:39.019 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T17:52:39.547 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T17:52:39.627 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T17:52:39.697 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T17:52:39.721 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:52:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:52:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T17:52:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T17:52:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T17:52:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T17:52:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T17:52:41.135 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:41.136 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:41.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-03-25T17:52:41.968 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:42.116 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:42.117 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-03-25T17:52:42.118 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 964 k 2024-03-25T17:52:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.8 M 2024-03-25T17:52:42.118 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:52:42.252 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 589 kB/s | 79 kB 00:00 2024-03-25T17:52:46.261 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 214 kB/s | 885 kB 00:04 2024-03-25T17:52:46.262 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:46.263 INFO:teuthology.orchestra.run.smithi090.stdout:Total 232 kB/s | 964 kB 00:04 2024-03-25T17:52:46.263 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:52:46.286 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:52:46.286 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:52:46.360 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:52:46.360 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:52:46.536 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:52:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T17:52:46.753 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:52:47.518 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:52:47.518 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:52:47.827 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T17:52:47.827 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:47.827 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:52:47.827 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T17:52:47.827 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:47.827 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:48.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-volume 2024-03-25T17:52:48.773 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:48.925 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:48.925 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-03-25T17:52:48.925 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:48.925 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-03-25T17:52:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:48.927 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 332 k 2024-03-25T17:52:48.927 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.5 M 2024-03-25T17:52:48.927 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:52:49.037 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 625 kB/s | 69 kB 00:00 2024-03-25T17:52:50.671 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 151 kB/s | 264 kB 00:01 2024-03-25T17:52:50.672 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:Total 190 kB/s | 332 kB 00:01 2024-03-25T17:52:50.673 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:52:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:52:50.682 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:52:50.729 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:52:50.730 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:52:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:52:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T17:52:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:52:51.115 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:52:51.115 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:52:51.115 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T17:52:51.116 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T17:52:52.276 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T17:52:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:52.277 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:52:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:52:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T17:52:52.277 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:52.277 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:52.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-03-25T17:52:53.310 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:08 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:53.461 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:53.462 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:52:53.463 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 127 k 2024-03-25T17:52:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 456 k 2024-03-25T17:52:53.463 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:52:54.274 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 156 kB/s | 127 kB 00:00 2024-03-25T17:52:54.275 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:54.275 INFO:teuthology.orchestra.run.smithi090.stdout:Total 156 kB/s | 127 kB 00:00 2024-03-25T17:52:54.276 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:52:54.282 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:52:54.282 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:52:54.342 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:52:54.342 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:52:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:52:54.589 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:55.120 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:55.426 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:55.426 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:55.426 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:52:55.426 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:55.426 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:55.426 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:55.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-03-25T17:52:56.301 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:56.403 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:56.458 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:56.459 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:52:56.459 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:56.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-03-25T17:52:57.099 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:12 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:57.248 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:57.248 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:57.248 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T17:52:57.248 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:57.248 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 31 k 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 140 k 2024-03-25T17:52:57.249 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:52:57.808 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 56 kB/s | 31 kB 00:00 2024-03-25T17:52:57.809 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:57.809 INFO:teuthology.orchestra.run.smithi090.stdout:Total 56 kB/s | 31 kB 00:00 2024-03-25T17:52:57.809 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:52:57.812 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:52:57.812 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:52:57.842 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:52:57.842 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:52:57.994 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:52:58.114 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:58.552 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:58.806 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:52:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:58.807 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:52:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:58.807 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:52:58.807 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:59.007 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-03-25T17:52:59.584 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:52:59.686 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:52:59.741 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:52:59.742 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:52:59.742 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:52:59.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-03-25T17:53:00.370 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:00.468 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:53:00.519 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:00.520 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:53:00.521 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:00.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-03-25T17:53:01.125 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:01.222 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:53:01.274 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:01.275 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:53:01.275 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:01.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-03-25T17:53:01.891 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:01.987 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:53:02.038 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:53:02.039 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:02.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-03-25T17:53:02.678 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:02.773 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:53:02.825 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:02.827 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:53:02.827 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:02.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-03-25T17:53:03.455 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:18 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:03.552 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:53:03.604 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:03.605 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T17:53:03.605 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:03.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-03-25T17:53:04.212 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:04.362 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:53:04.363 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:04.364 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 85 k 2024-03-25T17:53:04.364 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 231 k 2024-03-25T17:53:04.364 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:53:05.181 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 104 kB/s | 85 kB 00:00 2024-03-25T17:53:05.182 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:05.182 INFO:teuthology.orchestra.run.smithi090.stdout:Total 104 kB/s | 85 kB 00:00 2024-03-25T17:53:05.182 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:53:05.189 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:53:05.189 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:53:05.246 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:53:05.247 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:53:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:53:05.480 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:05.959 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:06.267 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:06.267 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:06.267 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:53:06.267 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:06.267 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:06.267 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:06.568 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-03-25T17:53:07.136 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:22 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:07.290 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:07.290 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:07.290 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:53:07.291 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:07.292 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.1 M 2024-03-25T17:53:07.292 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 13 M 2024-03-25T17:53:07.292 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:53:21.598 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 221 kB/s | 3.1 MB 00:14 2024-03-25T17:53:21.599 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:21.599 INFO:teuthology.orchestra.run.smithi090.stdout:Total 221 kB/s | 3.1 MB 00:14 2024-03-25T17:53:21.599 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:53:21.610 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:53:21.610 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:53:21.686 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:53:21.686 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:53:22.109 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:53:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:22.198 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:53:22.198 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T17:53:22.198 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:53:22.198 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:53:22.198 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:23.406 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:23.406 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:23.406 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:53:23.406 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:23.406 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:23.406 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:23.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-03-25T17:53:24.287 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:39 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:24.435 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:53:24.436 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:24.437 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 171 k 2024-03-25T17:53:24.437 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 494 k 2024-03-25T17:53:24.437 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:53:26.197 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 97 kB/s | 171 kB 00:01 2024-03-25T17:53:26.197 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:26.198 INFO:teuthology.orchestra.run.smithi090.stdout:Total 97 kB/s | 171 kB 00:01 2024-03-25T17:53:26.198 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:53:26.208 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:53:26.208 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:53:26.269 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:53:26.269 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:53:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:53:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:27.037 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:27.317 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:53:27.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:27.317 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:53:27.318 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:27.318 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:27.318 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:53:27.586 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:53:28.154 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:06:43 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:53:28.305 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:53:28.305 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T17:53:28.305 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:28.305 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:53:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:53:28.306 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 91 M 2024-03-25T17:53:28.307 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 498 M 2024-03-25T17:53:28.307 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:53:43.059 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 349 kB/s | 22 MB 01:05 2024-03-25T17:54:39.189 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 324 kB/s | 22 MB 01:10 2024-03-25T17:55:38.249 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 537 kB/s | 68 MB 02:09 2024-03-25T17:55:38.250 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:38.250 INFO:teuthology.orchestra.run.smithi090.stdout:Total 714 kB/s | 91 MB 02:09 2024-03-25T17:55:38.250 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T17:55:38.254 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T17:55:38.254 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T17:55:38.838 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T17:55:38.839 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T17:55:40.289 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 381 kB/s | 68 MB 03:03 2024-03-25T17:55:40.291 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:40.291 INFO:teuthology.orchestra.run.smithi060.stdout:Total 507 kB/s | 91 MB 03:03 2024-03-25T17:55:40.291 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T17:55:40.296 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T17:55:40.296 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T17:55:40.907 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T17:55:40.907 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T17:55:41.938 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T17:55:44.186 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T17:55:49.069 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:55:49.142 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:55:51.696 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:55:51.768 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:55:53.502 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:55:53.502 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T17:55:53.767 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:55:53.767 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:55:53.768 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T17:55:53.768 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:53.768 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:53.768 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:55:53.768 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T17:55:54.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-common-debuginfo 2024-03-25T17:55:54.611 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:09:09 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T17:55:54.759 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T17:55:54.759 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 416 M 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.7 G 2024-03-25T17:55:54.760 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T17:55:55.304 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:55:55.305 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T17:55:55.585 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:55:55.586 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:55:55.586 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T17:55:55.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:55.586 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:55.587 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:55:55.587 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T17:55:55.807 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-common-debuginfo 2024-03-25T17:55:56.379 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:09:30 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T17:55:56.532 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T17:55:56.533 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T17:55:56.534 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T17:55:56.534 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T17:55:56.534 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 416 M 2024-03-25T17:55:56.534 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.7 G 2024-03-25T17:55:56.534 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:16:46.514 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 340 kB/s | 416 MB 20:51 2024-03-25T18:16:46.515 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:46.515 INFO:teuthology.orchestra.run.smithi090.stdout:Total 340 kB/s | 416 MB 20:51 2024-03-25T18:16:46.515 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T18:16:46.518 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T18:16:46.518 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T18:16:48.986 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T18:16:48.986 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T18:17:31.036 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T18:17:31.132 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:17:37.550 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:17:37.802 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:17:37.803 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:37.803 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T18:17:37.803 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:37.803 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:37.803 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:17:38.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:17:38.672 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:30:53 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:17:38.820 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:17:38.820 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T18:17:38.820 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T18:17:38.820 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T18:17:38.820 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T18:17:38.820 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================== 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.4 M 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 6.4 M 2024-03-25T18:17:38.821 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T18:17:41.304 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 589 kB/s | 1.4 MB 00:02 2024-03-25T18:17:41.304 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:17:41.305 INFO:teuthology.orchestra.run.smithi090.stdout:Total 588 kB/s | 1.4 MB 00:02 2024-03-25T18:17:41.305 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T18:17:41.309 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T18:17:41.309 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T18:17:41.354 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T18:17:41.354 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T18:17:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T18:17:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:17:42.248 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:17:42.508 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:17:42.508 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:42.508 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T18:17:42.509 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:17:42.509 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:42.509 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:17:42.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:17:43.306 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:30:58 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:17:43.453 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T18:17:43.454 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:17:43.455 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 176 M 2024-03-25T18:17:43.455 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 753 M 2024-03-25T18:17:43.455 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T18:19:38.889 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 300 kB/s | 416 MB 23:42 2024-03-25T18:19:38.890 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:19:38.890 INFO:teuthology.orchestra.run.smithi060.stdout:Total 300 kB/s | 416 MB 23:42 2024-03-25T18:19:38.890 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T18:19:38.895 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T18:19:38.895 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T18:19:41.484 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T18:19:41.484 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T18:20:26.212 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T18:20:26.304 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:34.477 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:34.735 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:20:34.735 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:34.735 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T18:20:34.735 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:34.735 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:34.735 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:20:35.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:20:35.594 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:34:09 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:20:35.748 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:20:35.748 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T18:20:35.748 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T18:20:35.748 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T18:20:35.748 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================== 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:35.749 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 1.4 M 2024-03-25T18:20:35.750 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 6.4 M 2024-03-25T18:20:35.750 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:20:39.405 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 400 kB/s | 1.4 MB 00:03 2024-03-25T18:20:39.405 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:39.406 INFO:teuthology.orchestra.run.smithi060.stdout:Total 400 kB/s | 1.4 MB 00:03 2024-03-25T18:20:39.406 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T18:20:39.410 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T18:20:39.410 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T18:20:39.457 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T18:20:39.458 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T18:20:39.723 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T18:20:39.822 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:20:40.212 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:20:40.506 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:20:40.506 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:40.506 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T18:20:40.507 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:40.507 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:40.507 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:20:40.730 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:20:41.302 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:34:15 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:20:41.457 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:20:41.457 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:20:41.457 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T18:20:41.457 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:20:41.457 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T18:20:41.457 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:20:41.458 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:41.458 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T18:20:41.458 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:20:41.458 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T18:20:41.458 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:20:41.458 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 176 M 2024-03-25T18:20:41.458 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 753 M 2024-03-25T18:20:41.459 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:23:25.406 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 526 kB/s | 176 MB 05:41 2024-03-25T18:23:25.406 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:25.407 INFO:teuthology.orchestra.run.smithi090.stdout:Total 526 kB/s | 176 MB 05:41 2024-03-25T18:23:25.407 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T18:23:25.410 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T18:23:25.410 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T18:23:26.470 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T18:23:26.470 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T18:23:44.467 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T18:23:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:23:50.588 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:23:50.836 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:23:50.836 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:23:50.836 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T18:23:50.836 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:23:50.836 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:23:50.836 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:23:51.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:23:51.698 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:37:06 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:23:51.846 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:23:51.846 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:23:51.846 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T18:23:51.846 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:23:51.846 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T18:23:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:23:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:23:51.847 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T18:23:51.847 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:23:51.847 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T18:23:51.847 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:23:51.847 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 672 M 2024-03-25T18:23:51.848 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 3.1 G 2024-03-25T18:23:51.848 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T18:25:29.128 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 625 kB/s | 176 MB 04:47 2024-03-25T18:25:29.129 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:25:29.129 INFO:teuthology.orchestra.run.smithi060.stdout:Total 625 kB/s | 176 MB 04:47 2024-03-25T18:25:29.129 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T18:25:29.132 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T18:25:29.132 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T18:25:30.242 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T18:25:30.242 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T18:25:49.314 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T18:25:49.560 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:25:54.062 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:25:54.335 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T18:25:54.335 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:25:54.335 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T18:25:54.335 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:25:54.335 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:25:54.335 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:25:54.539 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test-debuginfo 2024-03-25T18:25:55.116 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:39:29 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:25:55.270 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:25:55.271 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:25:55.271 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T18:25:55.271 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:25:55.271 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T18:25:55.271 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T18:25:55.271 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:25:55.271 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T18:25:55.272 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:25:55.272 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T18:25:55.272 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:25:55.272 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 672 M 2024-03-25T18:25:55.272 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 3.1 G 2024-03-25T18:25:55.272 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:45:17.637 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 592 kB/s | 672 MB 19:22 2024-03-25T18:45:17.637 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:45:17.637 INFO:teuthology.orchestra.run.smithi060.stdout:Total 592 kB/s | 672 MB 19:22 2024-03-25T18:45:17.637 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T18:45:17.642 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T18:45:17.642 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T18:45:21.801 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T18:45:21.801 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T18:46:36.837 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T18:46:36.957 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:46:45.872 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:46:46.149 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:46:46.150 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:46:46.150 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T18:46:46.150 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:46:46.150 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:46:46.150 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:46:46.446 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:46:47.023 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:00:21 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:46:47.123 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:46:47.178 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:46:47.179 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-03-25T18:46:47.180 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:46:47.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:46:47.825 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:00:21 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:46:47.979 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:46:47.980 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:46:47.980 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T18:46:47.980 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:46:47.980 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T18:46:47.980 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:46:47.980 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:46:47.980 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T18:46:47.981 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:46:47.981 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T18:46:47.981 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:46:47.981 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 20 M 2024-03-25T18:46:47.981 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 103 M 2024-03-25T18:46:47.981 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:47:38.012 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 403 kB/s | 20 MB 00:50 2024-03-25T18:47:38.013 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:38.013 INFO:teuthology.orchestra.run.smithi060.stdout:Total 403 kB/s | 20 MB 00:50 2024-03-25T18:47:38.013 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T18:47:38.017 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T18:47:38.017 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T18:47:38.174 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T18:47:38.174 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T18:47:40.528 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T18:47:40.638 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:47:41.598 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:47:41.890 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:47:41.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:47:41.890 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T18:47:41.890 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:47:41.890 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:47:41.890 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:47:42.067 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:47:42.640 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:01:16 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:47:42.796 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T18:47:42.798 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:47:42.799 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 48 M 2024-03-25T18:47:42.799 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 173 M 2024-03-25T18:47:42.799 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:49:32.811 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 444 kB/s | 48 MB 01:50 2024-03-25T18:49:32.812 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:49:32.812 INFO:teuthology.orchestra.run.smithi060.stdout:Total 444 kB/s | 48 MB 01:50 2024-03-25T18:49:32.812 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T18:49:32.816 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T18:49:32.816 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T18:49:33.144 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T18:49:33.144 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T18:49:38.154 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T18:49:38.254 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:39.628 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:39.883 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:49:39.883 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:49:39.883 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T18:49:39.883 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:49:39.883 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:49:39.883 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:49:40.098 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:49:40.673 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:03:14 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:49:40.827 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:49:40.827 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:49:40.827 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T18:49:40.827 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:49:40.828 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T18:49:40.828 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:49:40.828 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:49:40.828 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T18:49:40.828 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:49:40.828 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T18:49:40.828 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:49:40.829 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 99 M 2024-03-25T18:49:40.829 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 449 M 2024-03-25T18:49:40.829 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:50:20.737 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 433 kB/s | 672 MB 26:28 2024-03-25T18:50:20.738 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:50:20.738 INFO:teuthology.orchestra.run.smithi090.stdout:Total 433 kB/s | 672 MB 26:28 2024-03-25T18:50:20.738 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T18:50:20.742 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T18:50:20.742 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T18:50:24.707 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T18:50:24.707 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T18:51:35.783 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T18:51:35.926 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:51:47.363 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:51:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:51:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:51:47.604 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T18:51:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:51:47.605 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:51:47.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:51:48.490 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:05:03 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:51:48.586 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:51:48.638 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:51:48.639 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-03-25T18:51:48.639 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:51:48.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:51:49.278 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:05:04 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:51:49.426 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T18:51:49.427 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:51:49.428 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T18:51:49.428 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:51:49.428 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 20 M 2024-03-25T18:51:49.428 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 103 M 2024-03-25T18:51:49.428 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T18:52:54.057 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 312 kB/s | 20 MB 01:04 2024-03-25T18:52:54.058 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:54.058 INFO:teuthology.orchestra.run.smithi090.stdout:Total 312 kB/s | 20 MB 01:04 2024-03-25T18:52:54.058 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T18:52:54.062 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T18:52:54.062 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T18:52:54.211 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T18:52:54.211 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T18:52:56.433 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T18:52:56.531 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:52:57.694 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:52:57.944 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:52:57.945 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:52:57.945 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T18:52:57.945 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:57.945 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:52:57.945 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:52:58.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:52:58.716 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:06:13 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:52:58.865 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:52:58.865 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:52:58.865 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T18:52:58.865 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:52:58.865 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T18:52:58.865 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:52:58.865 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:52:58.866 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T18:52:58.866 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:52:58.866 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T18:52:58.866 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:52:58.866 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 48 M 2024-03-25T18:52:58.866 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 173 M 2024-03-25T18:52:58.867 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T18:54:23.383 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 357 kB/s | 99 MB 04:42 2024-03-25T18:54:23.384 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:54:23.384 INFO:teuthology.orchestra.run.smithi060.stdout:Total 357 kB/s | 99 MB 04:42 2024-03-25T18:54:23.384 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T18:54:23.387 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T18:54:23.387 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T18:54:24.025 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T18:54:24.025 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T18:54:34.926 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T18:54:35.150 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:54:38.157 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:54:38.478 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:54:38.479 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:54:38.479 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T18:54:38.479 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:54:38.479 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:54:38.479 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T18:54:38.672 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:54:39.243 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:08:13 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T18:54:39.396 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T18:54:39.397 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:54:39.397 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T18:54:39.397 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:54:39.397 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 287 M 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.2 G 2024-03-25T18:54:39.398 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T18:55:44.951 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 294 kB/s | 48 MB 02:46 2024-03-25T18:55:44.952 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:55:44.952 INFO:teuthology.orchestra.run.smithi090.stdout:Total 294 kB/s | 48 MB 02:46 2024-03-25T18:55:44.952 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T18:55:44.955 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T18:55:44.955 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T18:55:45.267 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T18:55:45.267 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T18:55:50.048 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T18:55:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:51.797 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:55:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:55:52.045 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T18:55:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:55:52.045 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:55:52.045 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:55:52.242 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:55:52.802 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:09:07 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:55:52.952 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T18:55:52.953 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:55:52.954 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T18:55:52.954 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:55:52.954 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 99 M 2024-03-25T18:55:52.954 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 449 M 2024-03-25T18:55:52.954 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T18:59:34.461 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 455 kB/s | 99 MB 03:41 2024-03-25T18:59:34.462 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:59:34.462 INFO:teuthology.orchestra.run.smithi090.stdout:Total 455 kB/s | 99 MB 03:41 2024-03-25T18:59:34.462 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T18:59:34.464 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T18:59:34.465 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T18:59:35.073 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T18:59:35.073 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T18:59:45.344 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T18:59:45.445 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:59:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:59:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:59:49.435 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:59:49.436 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T18:59:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:59:49.436 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:59:49.436 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T18:59:49.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:59:50.230 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:13:05 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T18:59:50.379 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T18:59:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T18:59:50.381 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 287 M 2024-03-25T18:59:50.381 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.2 G 2024-03-25T18:59:50.381 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:06:45.783 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 404 kB/s | 287 MB 12:06 2024-03-25T19:06:45.783 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:06:45.783 INFO:teuthology.orchestra.run.smithi060.stdout:Total 404 kB/s | 287 MB 12:06 2024-03-25T19:06:45.783 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:06:45.787 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:06:45.787 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:06:47.577 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:06:47.577 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:07:18.725 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:07:18.831 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:26.449 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:07:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:26.771 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:07:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:26.771 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:26.771 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:07:27.003 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:07:27.579 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:21:01 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:07:27.734 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:07:27.735 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:27.735 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:07:27.735 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:27.735 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:07:27.735 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 5.9 M 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 28 M 2024-03-25T19:07:27.736 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:07:44.347 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 366 kB/s | 5.9 MB 00:16 2024-03-25T19:07:44.347 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:44.348 INFO:teuthology.orchestra.run.smithi060.stdout:Total 366 kB/s | 5.9 MB 00:16 2024-03-25T19:07:44.348 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:07:44.352 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:07:44.352 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:07:44.427 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:07:44.428 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:07:45.196 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:07:45.306 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:07:45.829 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:07:46.090 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:07:46.090 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:46.090 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:07:46.090 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:46.090 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:46.091 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:07:46.294 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:07:46.868 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:21:20 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:07:47.026 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:47.027 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:07:47.029 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:47.029 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 109 k 2024-03-25T19:07:47.029 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 440 k 2024-03-25T19:07:47.029 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:07:47.808 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 139 kB/s | 109 kB 00:00 2024-03-25T19:07:47.808 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:07:47.809 INFO:teuthology.orchestra.run.smithi060.stdout:Total 139 kB/s | 109 kB 00:00 2024-03-25T19:07:47.809 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:07:47.812 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:07:47.813 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:07:47.851 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:07:47.851 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:07:47.972 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:07:48.076 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:07:48.440 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:07:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:07:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:48.707 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:07:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:07:48.707 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:48.708 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:07:48.914 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:07:49.492 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:21:23 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:07:49.649 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:07:49.650 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:49.650 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:07:49.650 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:49.650 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 5.4 M 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 25 M 2024-03-25T19:07:49.651 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:08:08.159 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 296 kB/s | 5.4 MB 00:18 2024-03-25T19:08:08.159 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:08:08.160 INFO:teuthology.orchestra.run.smithi060.stdout:Total 296 kB/s | 5.4 MB 00:18 2024-03-25T19:08:08.160 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:08:08.164 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:08:08.164 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:08:08.235 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:08:08.235 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:08:08.941 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:08:09.051 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:08:09.696 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:08:09.958 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:08:09.958 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:08:09.958 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:08:09.958 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:08:09.958 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:08:09.958 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:08:10.300 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2-debuginfo 2024-03-25T19:08:10.878 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:21:44 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:08:11.033 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:08:11.034 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:08:11.035 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:08:11.035 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:08:11.035 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 44 M 2024-03-25T19:08:11.035 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 253 M 2024-03-25T19:08:11.035 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:10:03.317 INFO:teuthology.orchestra.run.smithi060.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 404 kB/s | 44 MB 01:52 2024-03-25T19:10:03.318 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:03.318 INFO:teuthology.orchestra.run.smithi060.stdout:Total 404 kB/s | 44 MB 01:52 2024-03-25T19:10:03.318 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:10:03.321 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:10:03.321 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:10:03.634 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:10:03.635 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:10:09.009 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:10:09.111 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:10:10.960 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:10:11.227 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:10:11.227 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:10:11.227 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:10:11.227 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:11.227 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:10:11.227 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:10:11.449 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:10:12.021 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:23:46 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:10:12.176 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:10:12.177 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:10:12.177 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:10:12.177 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:10:12.177 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:10:12.177 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:10:12.177 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:10:12.178 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:10:12.178 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:10:12.178 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:10:12.178 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:10:12.178 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 52 M 2024-03-25T19:10:12.178 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 308 M 2024-03-25T19:10:12.178 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:11:59.223 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 403 kB/s | 287 MB 12:08 2024-03-25T19:11:59.223 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:11:59.223 INFO:teuthology.orchestra.run.smithi090.stdout:Total 403 kB/s | 287 MB 12:08 2024-03-25T19:11:59.223 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:11:59.226 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:11:59.226 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:12:00.944 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:12:00.944 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:12:21.097 INFO:teuthology.orchestra.run.smithi060.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 412 kB/s | 52 MB 02:08 2024-03-25T19:12:21.098 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:21.098 INFO:teuthology.orchestra.run.smithi060.stdout:Total 412 kB/s | 52 MB 02:08 2024-03-25T19:12:21.099 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:12:21.102 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:12:21.102 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:12:21.463 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:12:21.463 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:12:27.821 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:12:27.932 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:12:30.207 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:12:30.357 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:12:30.473 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:12:30.473 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:30.473 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:12:30.473 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:30.473 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:30.473 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:12:30.661 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:12:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:31.232 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:26:05 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:12:31.386 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:12:31.387 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:31.388 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 615 k 2024-03-25T19:12:31.388 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.1 M 2024-03-25T19:12:31.388 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:12:32.583 INFO:teuthology.orchestra.run.smithi060.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 514 kB/s | 615 kB 00:01 2024-03-25T19:12:32.584 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:32.584 INFO:teuthology.orchestra.run.smithi060.stdout:Total 514 kB/s | 615 kB 00:01 2024-03-25T19:12:32.584 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:12:32.588 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:12:32.588 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:12:32.630 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:12:32.631 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:12:32.806 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:12:32.918 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:33.335 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:33.616 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:12:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:33.616 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:33.616 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:12:33.840 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:12:34.414 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:26:08 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:12:34.572 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:12:34.573 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:34.573 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 771 k 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.7 M 2024-03-25T19:12:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:12:35.501 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:35.771 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:35.771 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:35.771 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:12:35.771 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:35.771 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:35.771 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:12:36.005 INFO:teuthology.orchestra.run.smithi060.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 539 kB/s | 771 kB 00:01 2024-03-25T19:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout:Total 538 kB/s | 771 kB 00:01 2024-03-25T19:12:36.006 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:12:36.010 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:12:36.010 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:12:36.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:12:36.055 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:12:36.055 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:12:36.263 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:12:36.374 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:12:36.588 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:25:51 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:12:36.736 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:12:36.737 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:36.737 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:12:36.737 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:36.737 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:12:36.737 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 5.9 M 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 28 M 2024-03-25T19:12:36.738 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:12:36.762 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:12:37.030 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:12:37.030 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:37.030 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:12:37.030 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:37.030 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:37.030 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:12:37.265 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:12:37.840 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:26:11 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:12:37.995 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:12:37.995 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:12:37.995 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:37.995 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:12:37.995 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 1.1 M 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 4.3 M 2024-03-25T19:12:37.996 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:12:39.917 INFO:teuthology.orchestra.run.smithi060.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 599 kB/s | 1.1 MB 00:01 2024-03-25T19:12:39.917 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:39.918 INFO:teuthology.orchestra.run.smithi060.stdout:Total 599 kB/s | 1.1 MB 00:01 2024-03-25T19:12:39.918 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:12:39.922 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:12:39.922 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:12:39.968 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:12:39.969 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:12:40.205 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:12:40.302 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:12:40.691 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:12:40.971 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:12:40.971 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:40.971 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:12:40.971 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:40.971 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:40.971 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:12:41.202 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:12:41.777 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:26:15 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:12:41.932 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 337 k 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.0 M 2024-03-25T19:12:41.933 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:12:42.945 INFO:teuthology.orchestra.run.smithi060.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 334 kB/s | 337 kB 00:01 2024-03-25T19:12:42.945 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:42.946 INFO:teuthology.orchestra.run.smithi060.stdout:Total 333 kB/s | 337 kB 00:01 2024-03-25T19:12:42.946 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:12:42.950 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:12:42.950 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:12:42.993 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:12:42.993 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:12:43.160 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:12:43.264 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:12:43.665 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:12:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:12:43.928 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:43.929 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:12:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:43.929 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:43.929 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:12:44.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:12:44.720 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:26:18 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:12:44.875 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 509 k 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 2.5 M 2024-03-25T19:12:44.876 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:12:46.065 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 428 kB/s | 509 kB 00:01 2024-03-25T19:12:46.066 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:46.066 INFO:teuthology.orchestra.run.smithi060.stdout:Total 428 kB/s | 509 kB 00:01 2024-03-25T19:12:46.066 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:12:46.070 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:12:46.070 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:12:46.114 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:12:46.114 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:12:46.309 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:12:46.409 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:46.807 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:47.079 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:12:47.079 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:47.079 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:12:47.080 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:47.080 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:47.080 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:12:47.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:12:47.850 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:26:21 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:12:48.006 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:12:48.007 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:12:48.008 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 52 M 2024-03-25T19:12:48.008 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 298 M 2024-03-25T19:12:48.008 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:12:49.212 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 488 kB/s | 5.9 MB 00:12 2024-03-25T19:12:49.212 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:49.213 INFO:teuthology.orchestra.run.smithi090.stdout:Total 488 kB/s | 5.9 MB 00:12 2024-03-25T19:12:49.213 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:12:49.217 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:12:49.217 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:12:49.291 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:12:49.291 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:12:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:12:50.136 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:12:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:12:51.137 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:12:51.137 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:51.137 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:12:51.137 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:51.137 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:51.137 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:12:51.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:12:51.937 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:26:06 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:12:52.087 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:12:52.088 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:12:52.088 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 109 k 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 440 k 2024-03-25T19:12:52.089 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:12:52.858 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 141 kB/s | 109 kB 00:00 2024-03-25T19:12:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:12:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:Total 141 kB/s | 109 kB 00:00 2024-03-25T19:12:52.859 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:12:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:12:52.863 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:12:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:12:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:12:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:12:53.123 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:53.557 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:53.825 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:12:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:53.826 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:12:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:12:53.826 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:53.826 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:12:54.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:12:54.670 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:26:09 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:12:54.820 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:12:54.820 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:54.820 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:12:54.821 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:12:54.822 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 5.4 M 2024-03-25T19:12:54.822 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 25 M 2024-03-25T19:12:54.822 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:13:01.079 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 876 kB/s | 5.4 MB 00:06 2024-03-25T19:13:01.080 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:01.080 INFO:teuthology.orchestra.run.smithi090.stdout:Total 876 kB/s | 5.4 MB 00:06 2024-03-25T19:13:01.081 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:13:01.085 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:13:01.085 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:13:01.152 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:13:01.153 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:13:01.813 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:13:01.953 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:02.584 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:13:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:13:02.842 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:13:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:13:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:13:02.843 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:13:03.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2-debuginfo 2024-03-25T19:13:03.642 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:26:18 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:13:03.792 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:13:03.793 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:13:03.794 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 44 M 2024-03-25T19:13:03.794 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 253 M 2024-03-25T19:13:03.794 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:14:32.833 INFO:teuthology.orchestra.run.smithi090.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 509 kB/s | 44 MB 01:29 2024-03-25T19:14:32.834 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:14:32.834 INFO:teuthology.orchestra.run.smithi090.stdout:Total 509 kB/s | 44 MB 01:29 2024-03-25T19:14:32.834 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:14:32.837 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:14:32.837 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:14:33.136 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:14:33.136 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:14:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:14:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:14:40.591 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:14:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:14:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:14:40.854 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:14:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:14:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:14:40.855 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:14:41.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:14:41.663 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:27:56 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:14:41.822 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:14:41.823 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:14:41.823 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:14:41.823 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:14:41.823 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 52 M 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 308 M 2024-03-25T19:14:41.824 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:15:04.012 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 388 kB/s | 52 MB 02:16 2024-03-25T19:15:04.013 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:15:04.013 INFO:teuthology.orchestra.run.smithi060.stdout:Total 388 kB/s | 52 MB 02:16 2024-03-25T19:15:04.013 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:15:04.017 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:15:04.017 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:15:04.370 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:15:04.370 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:15:10.632 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:15:10.740 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:15:12.797 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:15:13.078 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:15:13.078 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:15:13.078 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:15:13.079 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:13.079 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:15:13.079 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:15:13.259 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:15:13.833 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 1:28:47 ago on Mon 25 Mar 2024 05:46:26 PM UTC. 2024-03-25T19:15:13.988 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-03-25T19:15:13.989 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:15:13.989 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 1.3 M 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 5.5 M 2024-03-25T19:15:13.990 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-03-25T19:15:17.048 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 419 kB/s | 1.3 MB 00:03 2024-03-25T19:15:17.048 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:15:17.049 INFO:teuthology.orchestra.run.smithi060.stdout:Total 419 kB/s | 1.3 MB 00:03 2024-03-25T19:15:17.049 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-03-25T19:15:17.053 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-03-25T19:15:17.053 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-03-25T19:15:17.101 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-03-25T19:15:17.101 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-03-25T19:15:17.355 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-03-25T19:15:17.450 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:15:17.860 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:15:18.134 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:15:18.134 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:15:18.134 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-03-25T19:15:18.134 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:15:18.134 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:15:18.135 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-03-25T19:15:18.360 DEBUG:teuthology.parallel:result is None 2024-03-25T19:17:02.222 INFO:teuthology.orchestra.run.smithi090.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 378 kB/s | 52 MB 02:20 2024-03-25T19:17:02.222 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:02.222 INFO:teuthology.orchestra.run.smithi090.stdout:Total 378 kB/s | 52 MB 02:20 2024-03-25T19:17:02.222 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:17:02.225 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:17:02.225 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:17:02.569 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:17:02.569 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:17:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:17:08.646 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:17:11.457 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:17:11.733 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:17:11.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:11.733 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:17:11.733 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:11.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:11.733 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:17:11.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:17:12.515 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:30:27 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:17:12.665 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:17:12.666 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 615 k 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.1 M 2024-03-25T19:17:12.667 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:17:16.718 INFO:teuthology.orchestra.run.smithi090.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 152 kB/s | 615 kB 00:04 2024-03-25T19:17:16.718 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:16.719 INFO:teuthology.orchestra.run.smithi090.stdout:Total 152 kB/s | 615 kB 00:04 2024-03-25T19:17:16.719 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:17:16.723 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:17:16.723 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:17:16.764 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:17:16.764 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:17:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:17:17.042 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:17:17.494 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:17:17.757 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:17:17.757 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:17.757 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:17:17.757 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:17.757 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:17.757 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:17:17.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:17:18.543 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:30:33 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:17:18.693 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:17:18.694 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:18.694 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:17:18.694 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:18.694 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:17:18.694 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:17:18.694 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:18.695 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:17:18.695 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:18.695 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:17:18.695 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:18.695 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 771 k 2024-03-25T19:17:18.695 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.7 M 2024-03-25T19:17:18.695 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:17:22.038 INFO:teuthology.orchestra.run.smithi090.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 231 kB/s | 771 kB 00:03 2024-03-25T19:17:22.039 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:22.039 INFO:teuthology.orchestra.run.smithi090.stdout:Total 230 kB/s | 771 kB 00:03 2024-03-25T19:17:22.039 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:17:22.043 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:17:22.043 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:17:22.086 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:17:22.086 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:17:22.265 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:17:22.370 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:17:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:17:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:17:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:23.057 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:17:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:23.057 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:17:23.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:17:23.850 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:30:38 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:17:24.000 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:17:24.000 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:17:24.001 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.1 M 2024-03-25T19:17:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 4.3 M 2024-03-25T19:17:24.002 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:17:27.710 INFO:teuthology.orchestra.run.smithi090.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 310 kB/s | 1.1 MB 00:03 2024-03-25T19:17:27.711 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:27.711 INFO:teuthology.orchestra.run.smithi090.stdout:Total 310 kB/s | 1.1 MB 00:03 2024-03-25T19:17:27.711 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:17:27.715 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:17:27.715 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:17:27.759 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:17:27.759 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:17:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:17:28.075 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:17:28.545 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:17:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:17:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:28.797 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:17:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:28.798 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:17:29.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:17:29.573 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:30:44 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:17:29.722 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:17:29.723 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:29.724 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 337 k 2024-03-25T19:17:29.724 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.0 M 2024-03-25T19:17:29.724 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:17:30.739 INFO:teuthology.orchestra.run.smithi090.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 332 kB/s | 337 kB 00:01 2024-03-25T19:17:30.740 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:30.740 INFO:teuthology.orchestra.run.smithi090.stdout:Total 332 kB/s | 337 kB 00:01 2024-03-25T19:17:30.741 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:17:30.744 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:17:30.744 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:17:30.783 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:17:30.784 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:17:30.921 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:17:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:17:31.469 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:17:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:17:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:31.733 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:17:31.733 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:31.734 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:31.734 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:17:31.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:17:32.520 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:30:47 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:17:32.669 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:17:32.670 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:32.670 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:17:32.670 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:32.670 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:17:32.670 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:17:32.670 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:17:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:17:32.671 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 509 k 2024-03-25T19:17:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.5 M 2024-03-25T19:17:32.671 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:17:33.886 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 419 kB/s | 509 kB 00:01 2024-03-25T19:17:33.887 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:33.887 INFO:teuthology.orchestra.run.smithi090.stdout:Total 418 kB/s | 509 kB 00:01 2024-03-25T19:17:33.888 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:17:33.891 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:17:33.891 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:17:33.931 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:17:33.932 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:17:34.094 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:17:34.194 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:17:34.648 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:17:34.915 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:17:34.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:34.915 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:17:34.915 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:34.915 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:34.916 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:17:35.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:17:35.702 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:30:50 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:17:35.851 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:17:35.852 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:17:35.853 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 52 M 2024-03-25T19:17:35.853 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 298 M 2024-03-25T19:17:35.853 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:19:11.849 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 550 kB/s | 52 MB 01:35 2024-03-25T19:19:11.850 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:11.851 INFO:teuthology.orchestra.run.smithi090.stdout:Total 550 kB/s | 52 MB 01:35 2024-03-25T19:19:11.851 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:19:11.854 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:19:11.854 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:19:12.187 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:19:12.188 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:19:18.068 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:19:18.164 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:19:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:19:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:19:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:21.085 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:19:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:21.085 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:21.085 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:19:21.305 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:19:21.860 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 1:32:36 ago on Mon 25 Mar 2024 05:46:45 PM UTC. 2024-03-25T19:19:22.009 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-03-25T19:19:22.009 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:19:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-03-25T19:19:22.009 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:19:22.009 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-03-25T19:19:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:19:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:22.010 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-03-25T19:19:22.010 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-03-25T19:19:22.010 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-03-25T19:19:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:22.010 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 1.3 M 2024-03-25T19:19:22.010 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 5.5 M 2024-03-25T19:19:22.010 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-03-25T19:19:23.886 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 683 kB/s | 1.3 MB 00:01 2024-03-25T19:19:23.887 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:19:23.887 INFO:teuthology.orchestra.run.smithi090.stdout:Total 683 kB/s | 1.3 MB 00:01 2024-03-25T19:19:23.888 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-03-25T19:19:23.892 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-03-25T19:19:23.892 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-03-25T19:19:23.936 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-03-25T19:19:23.937 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-03-25T19:19:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-03-25T19:19:24.277 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:19:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:19:25.052 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:19:25.052 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:25.052 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-03-25T19:19:25.052 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:19:25.052 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:25.052 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-03-25T19:19:25.282 DEBUG:teuthology.parallel:result is None 2024-03-25T19:19:25.283 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:19:25.424 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:19:25.447 INFO:teuthology.orchestra.run.smithi060.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:19:25.448 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:19:25.448 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:19:25.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:19:25.590 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:19:25.611 INFO:teuthology.orchestra.run.smithi090.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:19:25.611 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:19:25.611 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:19:25.612 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:19:25.612 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:25.613 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:19:25.644 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:25.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:19:25.678 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:19:25.678 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:25.678 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:19:25.715 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:19:25.784 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:25.784 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:19:25.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:19:25.878 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:19:25.879 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:25.879 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:19:25.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:19:25.977 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:25.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:19:26.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:19:26.070 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:19:26.070 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:26.070 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:19:26.099 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:19:26.167 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:26.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:19:26.193 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:19:26.259 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:19:26.338 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:19:26.338 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-03-25T19:19:26.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2024-03-25T19:19:26.367 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:19:26.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:19:26.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:19:26.436 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:19:26.437 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:19:26.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:19:26.530 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:19:26.563 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:19:26.563 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621299', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC8ogJIpy1TBt+IUhnJ7ZyvgSe3upjMvybdFt9GMEHJsjODab+wJCwQcVzxQfKN8/0bekXnyckYIX/8utXEO7xo=', 'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdJt5xwrd3mWp6ij5wkj+fiGIV0KjKbkFyUqxxutZYyHZkvAiIrk7XMRh7jrWfGA9SGzmoXz15JMe8waoLYMJY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:19:26.563 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:19:26.566 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:19:26.580 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:19:26.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:19:26.647 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:26.647 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:19:26.700 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:19:26.701 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-25 17:52:32.032542335 +0000 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-25 17:45:14.881229595 +0000 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-25 17:45:14.881229595 +0000 2024-03-25T19:19:26.755 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-25 17:45:14.881229595 +0000 2024-03-25T19:19:26.755 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:19:26.820 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-25T19:19:26.820 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-25T19:19:26.820 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 7.5312e-05 s, 6.8 MB/s 2024-03-25T19:19:26.821 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:19:26.876 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-03-25T19:19:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:19:26.930 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:26.930 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:19:26.930 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:26.931 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:26.931 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-25 17:52:32.033542324 +0000 2024-03-25T19:19:26.931 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-25 17:45:15.213226109 +0000 2024-03-25T19:19:26.931 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-25 17:45:15.213226109 +0000 2024-03-25T19:19:26.931 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-25 17:45:15.213226109 +0000 2024-03-25T19:19:26.931 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:19:26.994 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-25T19:19:26.994 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-25T19:19:26.995 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.3388e-05 s, 5.5 MB/s 2024-03-25T19:19:26.996 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:19:27.051 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-03-25T19:19:27.105 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:19:27.105 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:27.106 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:19:27.106 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:27.106 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:27.106 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-25 17:52:32.033542324 +0000 2024-03-25T19:19:27.106 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-25 17:45:15.542222655 +0000 2024-03-25T19:19:27.106 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-25 17:45:15.542222655 +0000 2024-03-25T19:19:27.106 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-25 17:45:15.542222655 +0000 2024-03-25T19:19:27.106 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:19:27.170 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-25T19:19:27.170 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-25T19:19:27.170 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.5769e-05 s, 5.3 MB/s 2024-03-25T19:19:27.171 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:19:27.226 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-03-25T19:19:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:19:27.280 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:27.280 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:19:27.280 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:27.281 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:27.281 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-03-25 17:52:32.033542324 +0000 2024-03-25T19:19:27.281 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-03-25 17:45:15.870219212 +0000 2024-03-25T19:19:27.281 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-03-25 17:45:15.870219212 +0000 2024-03-25T19:19:27.281 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-03-25 17:45:15.870219212 +0000 2024-03-25T19:19:27.281 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:19:27.345 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-03-25T19:19:27.345 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-03-25T19:19:27.345 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.404e-05 s, 5.4 MB/s 2024-03-25T19:19:27.346 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:19:27.401 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:27.401 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:27.401 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:19:27.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:19:27.417 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-25 17:53:22.452029777 +0000 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-25 17:45:08.416760289 +0000 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-25 17:45:08.416760289 +0000 2024-03-25T19:19:27.472 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-25 17:45:08.416760289 +0000 2024-03-25T19:19:27.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:19:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-25T19:19:27.538 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-25T19:19:27.539 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000215626 s, 2.4 MB/s 2024-03-25T19:19:27.540 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:19:27.596 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-25 17:53:22.452029777 +0000 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-25 17:45:08.758754656 +0000 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-25 17:45:08.758754656 +0000 2024-03-25T19:19:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-25 17:45:08.758754656 +0000 2024-03-25T19:19:27.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:19:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-25T19:19:27.716 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-25T19:19:27.717 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000179039 s, 2.9 MB/s 2024-03-25T19:19:27.717 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:19:27.774 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-03-25T19:19:27.828 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-25 17:53:22.453029760 +0000 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-25 17:45:09.089749204 +0000 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-25 17:45:09.089749204 +0000 2024-03-25T19:19:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-25 17:45:09.089749204 +0000 2024-03-25T19:19:27.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:19:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-25T19:19:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-25T19:19:27.894 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000178816 s, 2.9 MB/s 2024-03-25T19:19:27.895 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:19:27.951 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-03-25T19:19:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:19:28.005 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:19:28.005 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:19:28.006 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:19:28.006 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:19:28.006 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-03-25 17:53:22.453029760 +0000 2024-03-25T19:19:28.006 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-03-25 17:45:09.424743687 +0000 2024-03-25T19:19:28.006 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-03-25 17:45:09.424743687 +0000 2024-03-25T19:19:28.006 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-03-25 17:45:09.424743687 +0000 2024-03-25T19:19:28.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:19:28.072 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-03-25T19:19:28.072 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-03-25T19:19:28.072 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.00119878 s, 427 kB/s 2024-03-25T19:19:28.073 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:19:28.128 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:28.128 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:19:28.128 INFO:tasks.ceph:Generating config... 2024-03-25T19:19:28.129 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:19:28.129 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:19:28.130 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:19:28.130 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:19:28.130 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:19:28.130 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:19:28.130 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:19:28.130 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:19:28.130 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:19:28.130 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:19:28.130 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:19:28.130 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:19:28.130 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:19:28.130 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:19:28.130 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:19:28.130 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:19:28.131 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:19:28.131 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:19:28.131 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:19:28.131 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:19:28.131 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:19:28.131 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:19:28.131 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:19:28.131 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:19:28.132 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:19:28.133 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:19:28.133 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:19:28.133 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:19:28.133 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:19:28.133 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:19:28.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:19:28.189 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:19:28.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:19:28.236 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:19:28.263 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.60:6789]'), ('mon.c', '[v1:172.21.15.60:6790]'), ('mon.b', '[v1:172.21.15.90:6789]')] 2024-03-25T19:19:28.263 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.60:6789],[v1:172.21.15.60:6790],[v1:172.21.15.90:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:19:28.264 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:28.264 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:19:28.318 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.60:6789]' --addv c '[v1:172.21.15.60:6790]' --addv b '[v1:172.21.15.90:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:28.402 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:28.402 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:19:28.402 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = quincy 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 17 (quincy) 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 3 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:19:28.403 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:19:28.404 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:19:28.418 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 69068b48-8881-455b-b2ea-bfe0d20a1ddc... 2024-03-25T19:19:28.418 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:19:28.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-03-25T19:19:28.509 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-03-25T19:19:28.510 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.511 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = "[v1:172.21.15.60:6789],[v1:172.21.15.60:6790],[v1:172.21.15.90:6789]" 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: mon client directed command retry = 5 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: mon election default strategy = 3 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: ms bind msgr2 = False 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: ms inject socket failures = 5000 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: ms type = async 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: osd heartbeat grace = 80 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-03-25T19:19:28.512 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.513 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: bdev async discard = True 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: bdev enable discard = True 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore allocator = stupid 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore block size = 96636764160 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore fsck on mount = True 2024-03-25T19:19:28.514 INFO:teuthology.orchestra.run.smithi060.stdout: bluestore zero block detection = True 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 1 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 1 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: debug monc = 20 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 20 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 10 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd full ratio = 0.9 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: osd class default list = * 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: osd class load list = * 2024-03-25T19:19:28.515 INFO:teuthology.orchestra.run.smithi060.stdout: osd client watch timeout = 120 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify cached snaps = True 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify missing on start = True 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd fast shutdown = False 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd max markdown count = 1000 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock override recovery settings = True 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = bluestore 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd op queue = debug_random 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd op queue cut off = debug_random 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: osd shutdown pgref assert = True 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-03-25T19:19:28.516 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:19:28.517 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.518 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon min osdmap epochs = 50 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd crush smoke test = False 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune interval = 2 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune min = 15 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon scrub interval = 300 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on pool no app = False 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: paxos service trim min = 10 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: debug monc = 20 2024-03-25T19:19:28.519 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi060.stdout: debug objecter = 20 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi060.stdout: debug rados = 20 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.c] 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.b] 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2024-03-25T19:19:28.520 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:19:28.521 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:19:28.522 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "[v1:172.21.15.60:6789],[v1:172.21.15.60:6790],[v1:172.21.15.90:6789]" 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: mon client directed command retry = 5 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: mon election default strategy = 3 2024-03-25T19:19:28.523 INFO:teuthology.orchestra.run.smithi090.stdout: ms bind msgr2 = False 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: ms inject socket failures = 5000 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: ms type = async 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: osd heartbeat grace = 80 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 600 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2024-03-25T19:19:28.524 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug trim objects = true 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2024-03-25T19:19:28.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: bdev async discard = True 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: bdev enable discard = True 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore allocator = stupid 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore block size = 96636764160 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore fsck on mount = True 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore zero block detection = True 2024-03-25T19:19:28.526 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluefs = 1 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: debug bluestore = 1 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: debug monc = 20 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: debug rocksdb = 10 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd full ratio = 0.9 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: osd class default list = * 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: osd class load list = * 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: osd client watch timeout = 120 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify cached snaps = True 2024-03-25T19:19:28.527 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify missing on start = True 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd fast shutdown = False 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd max markdown count = 1000 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock override recovery settings = True 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = bluestore 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue = debug_random 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue cut off = debug_random 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: osd shutdown pgref assert = True 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2024-03-25T19:19:28.528 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2024-03-25T19:19:28.529 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:19:28.530 INFO:teuthology.orchestra.run.smithi090.stdout: mon min osdmap epochs = 50 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd crush smoke test = False 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune interval = 2 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune min = 15 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: mon scrub interval = 300 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on pool no app = False 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: paxos service trim min = 10 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:19:28.531 INFO:teuthology.orchestra.run.smithi090.stdout: debug monc = 20 2024-03-25T19:19:28.532 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2024-03-25T19:19:28.532 INFO:teuthology.orchestra.run.smithi090.stdout: debug objecter = 20 2024-03-25T19:19:28.532 INFO:teuthology.orchestra.run.smithi090.stdout: debug rados = 20 2024-03-25T19:19:28.532 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2024-03-25T19:19:28.532 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2024-03-25T19:19:28.532 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2024-03-25T19:19:28.533 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:19:28.533 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:19:28.587 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:19:28.588 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:28.588 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:19:28.602 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:28.602 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:19:28.656 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:19:28.656 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:28.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:19:28.657 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:19:28.735 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:28.735 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:28.790 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:19:28.790 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:28.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:19:28.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:19:28.829 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:28.830 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:28.885 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:19:28.885 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:19:28.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:19:28.939 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:19:28.941 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:19:28.999 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:19:29.002 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:19:29.002 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:19:29.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:19:29.044 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:19:29.059 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:19:29.101 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:19:29.116 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:19:29.116 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:19:29.116 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:19:29.143 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:29.143 INFO:tasks.ceph:role: osd.0 2024-03-25T19:19:29.143 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:19:29.143 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:29.210 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:29.211 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:29.242 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-25T19:19:29.244 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-25T19:19:29.244 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:19:29.314 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:19:29.389 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:19:29.456 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:29.456 INFO:tasks.ceph:role: osd.1 2024-03-25T19:19:29.456 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:19:29.457 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:29.522 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:29.555 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-25T19:19:29.556 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-25T19:19:29.557 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:19:29.631 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:19:29.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:19:29.771 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:29.771 INFO:tasks.ceph:role: osd.2 2024-03-25T19:19:29.771 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:19:29.772 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:19:29.837 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:29.837 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:29.838 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:29.869 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-25T19:19:29.871 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-25T19:19:29.872 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:19:29.942 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:19:30.014 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:19:30.081 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:30.081 INFO:tasks.ceph:role: osd.3 2024-03-25T19:19:30.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:19:30.081 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:19:30.148 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:30.149 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:30.181 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-03-25T19:19:30.183 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-03-25T19:19:30.183 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:19:30.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:19:30.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:30.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:30.416+0000 7f825d817640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:19:30.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:30.416+0000 7f825d817640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:19:30.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:30.416+0000 7f825d817640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:19:30.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:30.416+0000 7f825d817640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:19:30.416 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:30.416+0000 7f825d817640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:19:30.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:30.417+0000 7f825d817640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:19:33.261 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:19:33.292 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:33.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:33.384+0000 7f3895cd9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:19:33.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:33.384+0000 7f3895cd9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:19:33.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:33.384+0000 7f3895cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:19:33.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:33.384+0000 7f3895cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:19:33.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:33.384+0000 7f3895cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:19:33.384 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:33.384+0000 7f3895cd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:19:36.223 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:19:36.253 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:36.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:36.343+0000 7f9dd7008640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:19:36.343 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:36.344+0000 7f9dd7008640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:19:36.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:36.344+0000 7f9dd7008640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:19:36.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:36.344+0000 7f9dd7008640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:19:36.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:36.344+0000 7f9dd7008640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:19:36.344 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:36.344+0000 7f9dd7008640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:19:39.181 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:19:39.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:39.303+0000 7f9d04a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:19:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:39.303+0000 7f9d04a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:19:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:39.303+0000 7f9d04a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:19:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:39.303+0000 7f9d04a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:19:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:39.303+0000 7f9d04a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:19:39.303 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:39.303+0000 7f9d04a11640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:19:42.142 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:19:42.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:42.204 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:42.204 INFO:tasks.ceph:role: osd.4 2024-03-25T19:19:42.204 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:19:42.204 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:42.281 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:42.282 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:42.799 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-25T19:19:42.801 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-25T19:19:42.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:42.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:42.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:43.066 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:43.066 INFO:tasks.ceph:role: osd.5 2024-03-25T19:19:43.067 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:19:43.067 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:19:43.142 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:43.142 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:43.142 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:43.143 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:43.143 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:43.143 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:43.143 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:43.143 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:43.143 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:43.143 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:43.595 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-25T19:19:43.597 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-25T19:19:43.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:43.700 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:43.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:43.802 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:43.802 INFO:tasks.ceph:role: osd.6 2024-03-25T19:19:43.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:19:43.802 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:19:43.877 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:43.878 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:44.365 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-25T19:19:44.367 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-25T19:19:44.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:44.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:44.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:44.621 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:19:44.621 INFO:tasks.ceph:role: osd.7 2024-03-25T19:19:44.622 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:19:44.622 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:19:44.697 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:19:44.698 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:19:45.184 INFO:teuthology.orchestra.run.smithi090.stdout:Discarding blocks...Done. 2024-03-25T19:19:45.186 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2024-03-25T19:19:45.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:45.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:45.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:45.447 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:45.448+0000 7fd3e0622640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:19:45.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:45.448+0000 7fd3e0622640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:19:45.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:45.448+0000 7fd3e0622640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:19:45.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:45.448+0000 7fd3e0622640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:19:45.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:45.448+0000 7fd3e0622640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:19:45.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:45.449+0000 7fd3e0622640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:19:48.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:19:48.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:48.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:48.407+0000 7f0442611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:19:48.406 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:48.407+0000 7f0442611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:19:48.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:48.407+0000 7f0442611640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:19:48.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:48.407+0000 7f0442611640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:19:48.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:48.407+0000 7f0442611640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:19:48.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:48.407+0000 7f0442611640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:19:51.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:19:51.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:51.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:51.367+0000 7f89f78ae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:19:51.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:51.367+0000 7f89f78ae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:19:51.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:51.367+0000 7f89f78ae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:19:51.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:51.368+0000 7f89f78ae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:19:51.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:51.368+0000 7f89f78ae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:19:51.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:51.368+0000 7f89f78ae640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:19:54.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:19:54.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:54.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:54.335+0000 7f5a92c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:19:54.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:54.335+0000 7f5a92c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:19:54.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:54.335+0000 7f5a92c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:19:54.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:54.335+0000 7f5a92c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:19:54.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:54.335+0000 7f5a92c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:19:54.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:19:54.336+0000 7f5a92c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:19:57.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:19:57.207 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:19:57.207 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:57.207 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:19:57.238 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:57.238 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:19:57.308 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:57.308 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:19:57.377 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:57.377 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:19:57.445 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:57.445 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:19:57.513 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:57.513 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:19:57.544 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:57.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:19:57.615 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:57.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:19:57.686 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:57.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:19:57.755 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:57.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:19:57.823 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:19:57.823 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:19:57.840 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:19:57.840 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:19:57.880 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:19:57.880 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:19:57.882 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:19:57.923 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.y] 2024-03-25T19:19:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBAzgFm9DcOOBAA65xvUaxYv7nYM5g7l7X9oQ== 2024-03-25T19:19:57.923 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-03-25T19:19:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBCzgFmPC7eGBAAfz0BWzqQjHexo4b2yxmi6w== 2024-03-25T19:19:57.923 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-03-25T19:19:57.923 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBFzgFmB23yFhAAJLiVNrYUWV79XXxJlweKig== 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBIzgFmco+LFBAACjmgGF81AVRJM/TdZJ6MQw== 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBLzgFmBi0dEhAAm4mOdct8/76Fpepy4w0bxw== 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBBzgFmVdwMABAAK6tdHVa3Lt4BWZQV3+RlxA== 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.4] 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBRzgFmKCDAGhAAwlhMpP8SIfMrektNmzQXFQ== 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.5] 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBUzgFm4ERHGBAAde5zg25KGBdGraTyyYNJjA== 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.6] 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBXzgFm5iDuFRAAgnXXMcLKiBYdSdqlAHtubg== 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.7] 2024-03-25T19:19:57.924 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBazgFmJ4oCFBAAfZw2XXWAlSzJ1oAakxljYQ== 2024-03-25T19:19:57.925 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-03-25T19:19:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBBzgFm8Sy8AhAAuGdEsERtwgwxowzqqPyoMA== 2024-03-25T19:19:57.925 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2024-03-25T19:19:57.925 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBBzgFmWKkeBhAATNxpPFLc2lLCBkmxQgH8Tw== 2024-03-25T19:19:57.946 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.y] 2024-03-25T19:19:57.946 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBAzgFm9DcOOBAA65xvUaxYv7nYM5g7l7X9oQ== 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBCzgFmPC7eGBAAfz0BWzqQjHexo4b2yxmi6w== 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBFzgFmB23yFhAAJLiVNrYUWV79XXxJlweKig== 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBIzgFmco+LFBAACjmgGF81AVRJM/TdZJ6MQw== 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBLzgFmBi0dEhAAm4mOdct8/76Fpepy4w0bxw== 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBBzgFmVdwMABAAK6tdHVa3Lt4BWZQV3+RlxA== 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBRzgFmKCDAGhAAwlhMpP8SIfMrektNmzQXFQ== 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2024-03-25T19:19:57.947 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBUzgFm4ERHGBAAde5zg25KGBdGraTyyYNJjA== 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.6] 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBXzgFm5iDuFRAAgnXXMcLKiBYdSdqlAHtubg== 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.7] 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBazgFmJ4oCFBAAfZw2XXWAlSzJ1oAakxljYQ== 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBBzgFm8Sy8AhAAuGdEsERtwgwxowzqqPyoMA== 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-03-25T19:19:57.948 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBBzgFmWKkeBhAATNxpPFLc2lLCBkmxQgH8Tw== 2024-03-25T19:19:57.949 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:57.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:58.042 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.096 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.150 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.256 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:58.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:19:58.310 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.363 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.365 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.416 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.418 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.469 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:19:58.523 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:58.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:58.579 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:58.581 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:19:58.635 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:19:58.635 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:19:58.661 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:19:58.757 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:19:58.784 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:19:58.851 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:19:58.948 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:19:58.974 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:19:59.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:19:59.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:19:59.123 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:59.125 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:19:59.177 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:19:59.177 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:19:59.178 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:19:59.178 DEBUG:teuthology.orchestra.run.smithi060:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:19:59.180 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:19:59.180 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:19:59.180 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:19:59.180 DEBUG:teuthology.orchestra.run.smithi060:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:19:59.182 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:19:59.182 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:19:59.182 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:19:59.182 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:19:59.219 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:19:59.219 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:19:59.219 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:19:59.220 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:19:59.222 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:19:59.222 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:19:59.222 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:19:59.224 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:19:59.224 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:19:59.224 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.325+0000 7fbefb4cf640 10 monclient: get_monmap_and_config 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.325+0000 7fbefb4cf640 10 monclient: build_initial_monmap 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.325+0000 7fbefb4cf640 10 monclient: monmap: 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:59.327469+0000 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:59.327469+0000 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:19:59.326 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:19:59.327 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:19:59.327 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:19:59.327 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:19:59.327 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:19:59.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.326+0000 7fbefb4cf640 1 Processor -- start 2024-03-25T19:19:59.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.326+0000 7fbefb4cf640 1 -- start start 2024-03-25T19:19:59.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.326+0000 7fbefb4cf640 10 monclient: init 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 10 monclient(hunting): picked mon.noname-b con 0x7fbef40a9920 addr v1:172.21.15.60:6790/0 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbef40b45a0 addr v1:172.21.15.90:6789/0 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbef4146450 addr v1:172.21.15.60:6789/0 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef406b100 con 0x7fbef4146450 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40b2940 con 0x7fbef40b45a0 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef413fec0 con 0x7fbef40a9920 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbefb4cf640 10 monclient(hunting): authenticate will time out at 6326.426130s 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbef414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbef413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:19:59.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.327+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbef4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:19:59.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.528+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbef413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:19:59.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.528+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbef414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:19:59.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.528+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbef4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:19:59.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.929+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbef413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:19:59.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.929+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbef414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:19:59.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:19:59.929+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbef4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:00.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:00.730+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbef414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:00.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:00.730+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbef413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:00.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:00.730+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbef4145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:02.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient(hunting): tick 2024-03-25T19:20:02.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:20:02.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:20:02.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:20:02.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbef413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:02.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbef4145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbef414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fbef40a9920 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fbef40b45a0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fbef4146450 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef406b100 con 0x7fbef4146450 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40b2940 con 0x7fbef40b45a0 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef413fec0 con 0x7fbef40a9920 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbeeaffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:02.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.327+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:02.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.528+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.528+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:02.529 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.528+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:02.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.929+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:02.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.929+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:02.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:02.929+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:03.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:03.730+0000 7fbef926b640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:03.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:03.730+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:03.731 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:03.730+0000 7fbef8a6a640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient(hunting): tick 2024-03-25T19:20:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:20:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:20:05.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fbef40b45a0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fbef4146450 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fbef40a9920 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8010e20 con 0x7fbef4146450 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef406b100 con 0x7fbef40b45a0 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40b2940 con 0x7fbef40a9920 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbeeaffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbef8a6a640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:05.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.327+0000 7fbef926b640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.528+0000 7fbef8a6a640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.528+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:05.528 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.528+0000 7fbef926b640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:05.929 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.929+0000 7fbef8a6a640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.929+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:05.930 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:05.929+0000 7fbef926b640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:06.230 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:00:06.883 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:06.535 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:00:07.188 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:06.698 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:00:07.409 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:06.699 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:00:07.410 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:06.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:06.730+0000 7fbef8a6a640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6789/0 2024-03-25T19:20:06.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:06.730+0000 7fbef9a6c640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.60:6790/0 2024-03-25T19:20:06.730 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:06.730+0000 7fbef926b640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.90:6789/0 2024-03-25T19:20:07.024 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:00:07.736 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:07.025 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:00:07.737 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:20:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient(hunting): tick 2024-03-25T19:20:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:20:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:20:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:20:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:08.328 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed800fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 1 -- >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fbef4146450 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fbef40b45a0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fbef40a9920 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef413fec0 con 0x7fbef4146450 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8010e20 con 0x7fbef40b45a0 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef406b100 con 0x7fbef40a9920 2024-03-25T19:20:08.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.327+0000 7fbeeaffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.376+0000 7fbef8a6a640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 0x7fbed800fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:41016/0 (socket says 172.21.15.60:41016) 2024-03-25T19:20:08.376 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.376+0000 7fbef8a6a640 1 -- 172.21.15.60:0/2031075274 learned_addr learned my addr 172.21.15.60:0/2031075274 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:08.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.691+0000 7fbef9a6c640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T19:20:08.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.691+0000 7fbef9a6c640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:20:08.691 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.691+0000 7fbef9a6c640 1 --1- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 0x7fbed80056d0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T19:20:08.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.800+0000 7fbef926b640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:20:08.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.800+0000 7fbef926b640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:20:08.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.800+0000 7fbef926b640 1 --1- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 0x7fbed800c660 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T19:20:08.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.800+0000 7fbef926b640 1 --1- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 0x7fbed800c660 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:20:08.800 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:08.800+0000 7fbeebfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.60:6789/0 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient(hunting): tick 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6790/0 conn(0x7fbef40a9920 legacy=0x7fbed80056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed800fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fbed8009f20 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:11.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fbef40b45a0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fbef4146450 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40b2940 con 0x7fbef4146450 2024-03-25T19:20:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed800c400 con 0x7fbef40b45a0 2024-03-25T19:20:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8004720 con 0x7fbed8009f20 2024-03-25T19:20:11.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:11.328+0000 7fbeeaffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient(hunting): tick 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6790/0 conn(0x7fbed8009f20 legacy=0x7fbed8010620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.90:6789/0 conn(0x7fbef40b45a0 legacy=0x7fbed800fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbef4146450 legacy=0x7fbed800c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:14.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fbed80048b0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:14.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fbed8009f20 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:14.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fbed8010620 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:14.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8005980 con 0x7fbed8010620 2024-03-25T19:20:14.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8006ba0 con 0x7fbed80048b0 2024-03-25T19:20:14.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8005570 con 0x7fbed8009f20 2024-03-25T19:20:14.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:14.329+0000 7fbeeaffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient(hunting): tick 2024-03-25T19:20:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:20:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 1 monclient(hunting): continuing hunt 2024-03-25T19:20:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:20:17.329 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6790/0 conn(0x7fbed8009f20 legacy=0x7fbed800fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.90:6789/0 conn(0x7fbed80048b0 legacy=0x7fbed8004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbed8010620 legacy=0x7fbed8008e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fbed800c640 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fbed8011070 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fbed8004cf0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8005700 con 0x7fbed8004cf0 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed800cbc0 con 0x7fbed8011070 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed8005170 con 0x7fbed800c640 2024-03-25T19:20:17.330 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.328+0000 7fbeeaffd640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.565+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219386556 0 0) 0x7fbed8005700 con 0x7fbed8004cf0 2024-03-25T19:20:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.565+0000 7fbeebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.565+0000 7fbeebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.565+0000 7fbeebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.565+0000 7fbeebfff640 10 monclient(hunting): my global_id is 4104 2024-03-25T19:20:17.566 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.565+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee0003520 con 0x7fbed8004cf0 2024-03-25T19:20:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.605+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073904623 0 0) 0x7fbed800cbc0 con 0x7fbed8011070 2024-03-25T19:20:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.605+0000 7fbeebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.605+0000 7fbeebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.605+0000 7fbeebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.605+0000 7fbeebfff640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:20:17.606 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.605+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed8005700 con 0x7fbed8011070 2024-03-25T19:20:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.607+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550336914 0 0) 0x7fbed8005170 con 0x7fbed800c640 2024-03-25T19:20:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.607+0000 7fbeebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.607+0000 7fbeebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.607+0000 7fbeebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:17.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.607+0000 7fbeebfff640 10 monclient(hunting): my global_id is 4100 2024-03-25T19:20:17.608 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.607+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed800cbc0 con 0x7fbed800c640 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3679417387 0 0) 0x7fbee0003520 con 0x7fbed8004cf0 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6790/0 conn(0x7fbed800c640 legacy=0x7fbed8009550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.90:6789/0 conn(0x7fbed8011070 legacy=0x7fbed800fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 1 monclient: found mon.noname-a 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed8013470 con 0x7fbed8004cf0 2024-03-25T19:20:17.627 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 10 monclient: _finish_auth 0 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbeec004330 con 0x7fbed8004cf0 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbefb4cf640 5 monclient: authenticate success, global_id 4104 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbefb4cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbeebfff640 10 monclient: dump: 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:17.628 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:17.629 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:17.629 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.626+0000 7fbefb4cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbeec004490 con 0x7fbed8004cf0 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbeebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbeebfff640 1 -- 172.21.15.60:0/2031075274 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbeec0048a0 con 0x7fbed8004cf0 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbeebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbeebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbeebfff640 10 monclient: dump: 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:17.658 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbefb4cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:17.659 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.657+0000 7fbefb4cf640 4 set_mon_vals no callback set 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.658+0000 7fbefb4cf640 10 monclient: shutdown 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.658+0000 7fbefb4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.658+0000 7fbefb4cf640 1 -- 172.21.15.60:0/2031075274 >> v1:172.21.15.60:6789/0 conn(0x7fbed8004cf0 legacy=0x7fbed8008e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.658+0000 7fbefb4cf640 1 -- 172.21.15.60:0/2031075274 shutdown_connections 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.658+0000 7fbefb4cf640 1 -- 172.21.15.60:0/2031075274 >> 172.21.15.60:0/2031075274 conn(0x7fbef40b7f40 msgr2=0x7fbef40b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.658+0000 7fbefb4cf640 1 -- 172.21.15.60:0/2031075274 shutdown_connections 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.658+0000 7fbefb4cf640 1 -- 172.21.15.60:0/2031075274 wait complete. 2024-03-25T19:20:17.661 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 10 monclient: build_initial_monmap 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 10 monclient: monmap: 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 1 librados: starting msgr at 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 1 librados: starting objecter 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 1 Processor -- start 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 1 -- start start 2024-03-25T19:20:17.662 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 1 librados: setting wanted keys 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 1 librados: calling monclient init 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.659+0000 7fbefb4cf640 10 monclient: init 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbed8004cf0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 10 monclient(hunting): picked mon.noname-b con 0x7fbed800c640 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbed8011070 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40ad360 con 0x7fbed8004cf0 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef41da810 con 0x7fbed800c640 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40ac290 con 0x7fbed8011070 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.660+0000 7fbefb4cf640 10 monclient(hunting): authenticate will time out at 6344.759279s 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.661+0000 7fbef8a6a640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbed800c640 0x7fbef41dad30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:35818/0 (socket says 172.21.15.60:35818) 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.661+0000 7fbef8a6a640 1 -- 172.21.15.60:0/1173212045 learned_addr learned my addr 172.21.15.60:0/1173212045 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.661+0000 7fbee9ffb640 10 client.?.objecter ms_handle_connect 0x7fbed800c640 2024-03-25T19:20:17.663 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.661+0000 7fbee9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:17.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.662+0000 7fbee9ffb640 10 client.?.objecter ms_handle_connect 0x7fbed8011070 2024-03-25T19:20:17.664 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.662+0000 7fbee9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:17.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.713+0000 7fbee9ffb640 10 client.?.objecter ms_handle_connect 0x7fbed8004cf0 2024-03-25T19:20:17.713 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.713+0000 7fbee9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:17.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.719+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009975401 0 0) 0x7fbef40ad360 con 0x7fbed8004cf0 2024-03-25T19:20:17.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.719+0000 7fbee9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:17.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.719+0000 7fbee9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:17.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.719+0000 7fbee9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:17.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.719+0000 7fbee9ffb640 10 monclient(hunting): my global_id is 4107 2024-03-25T19:20:17.720 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.719+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed8005170 con 0x7fbed8004cf0 2024-03-25T19:20:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.737+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839963733 0 0) 0x7fbed8005170 con 0x7fbed8004cf0 2024-03-25T19:20:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.737+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbef40ad360 con 0x7fbed8004cf0 2024-03-25T19:20:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.737+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbef0004070 con 0x7fbed8004cf0 2024-03-25T19:20:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.737+0000 7fbee9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.737+0000 7fbee9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.737+0000 7fbee9ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:17.738 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:17.739 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.737+0000 7fbee9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.742+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247246407 0 0) 0x7fbef40ac290 con 0x7fbed8011070 2024-03-25T19:20:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.742+0000 7fbee9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.742+0000 7fbee9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.742+0000 7fbee9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.742+0000 7fbee9ffb640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:20:17.742 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.742+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed8005170 con 0x7fbed8011070 2024-03-25T19:20:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.744+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393646913 0 0) 0x7fbef41da810 con 0x7fbed800c640 2024-03-25T19:20:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.744+0000 7fbee9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.744+0000 7fbee9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.744+0000 7fbee9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.744+0000 7fbee9ffb640 10 monclient(hunting): my global_id is 4108 2024-03-25T19:20:17.745 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.744+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef40ac290 con 0x7fbed800c640 2024-03-25T19:20:17.758 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3503909925 0 0) 0x7fbef40ad360 con 0x7fbed8004cf0 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 >> v1:172.21.15.60:6790/0 conn(0x7fbed8011070 legacy=0x7fbef4138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 >> v1:172.21.15.90:6789/0 conn(0x7fbed800c640 legacy=0x7fbef41dad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 1 monclient: found mon.a 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef41db440 con 0x7fbed8004cf0 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbee9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 5 monclient: authenticate success, global_id 4107 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 10 monclient: _renew_subs 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef41449e0 con 0x7fbed8004cf0 2024-03-25T19:20:17.759 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 10 monclient: _renew_subs 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef4141ad0 con 0x7fbed8004cf0 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.758+0000 7fbefb4cf640 1 librados: init done 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.759+0000 7fbefb4cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.759+0000 7fbefb4cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.759+0000 7fbefb4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:17.760 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.759+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef40ad1a0 con 0x7fbed8004cf0 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef0004480 con 0x7fbed8004cf0 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbef0004890 con 0x7fbed8004cf0 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 10 monclient: dump: 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:17.990 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbef0004e40 con 0x7fbed8004cf0 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbeeb7fe640 4 set_mon_vals no callback set 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 848262956 0 0) 0x7fbef0006370 con 0x7fbed8004cf0 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 10 client.4107.objecter ms_dispatch 0x7fbef4000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbef0006740 con 0x7fbed8004cf0 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:17.991 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:17.989+0000 7fbee9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.057+0000 7fbefb4cf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:20:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.057+0000 7fbefb4cf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:20:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.057+0000 7fbefb4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:18.058 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.057+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fbef40ad1a0 con 0x7fbed8004cf0 2024-03-25T19:20:18.237 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:18.237+0000 7faf1d632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:20:18.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.247+0000 7fbee9ffb640 1 -- 172.21.15.60:0/1173212045 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fbef0005400 con 0x7fbed8004cf0 2024-03-25T19:20:18.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.247+0000 7fbee9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:20:18.248 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.247+0000 7fbee9ffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:20:18.248 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-03-25T19:20:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 10 librados: watch_flush enter 2024-03-25T19:20:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 10 librados: watch_flush exit 2024-03-25T19:20:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T19:20:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 10 client.4107.objecter successfully canceled tick 2024-03-25T19:20:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 10 monclient: shutdown 2024-03-25T19:20:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:18.249 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 >> v1:172.21.15.60:6789/0 conn(0x7fbed8004cf0 legacy=0x7fbef40abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:18.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 shutdown_connections 2024-03-25T19:20:18.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 >> 172.21.15.60:0/1173212045 conn(0x7fbef40b7f40 msgr2=0x7fbef40b76f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:18.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 shutdown_connections 2024-03-25T19:20:18.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 1 -- 172.21.15.60:0/1173212045 wait complete. 2024-03-25T19:20:18.250 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:18.248+0000 7fbefb4cf640 1 librados: shutdown 2024-03-25T19:20:18.260 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:20:18.260 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:20:18.260 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:20:18.260 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:20:18.287 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:20:18.287 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:20:18.292 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:18.292+0000 7faf1d632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:20:18.355 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:20:18.355 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:20:18.422 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-03-25T19:20:18.422 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:20:18.489 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:20:18.489 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:20:18.517 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:20:18.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:20:18.571 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:18.571+0000 7fd81362b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:20:18.583 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:20:18.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:20:18.629 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:18.629+0000 7fd81362b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:20:18.650 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-03-25T19:20:18.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:20:18.718 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 63dcc61d-0867-4284-a895-94136baa98f9 0 2024-03-25T19:20:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.858+0000 7f896843e640 10 monclient: get_monmap_and_config 2024-03-25T19:20:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.858+0000 7f896843e640 10 monclient: build_initial_monmap 2024-03-25T19:20:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.858+0000 7f896843e640 10 monclient: monmap: 2024-03-25T19:20:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:18.862 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:18.859565+0000 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:18.859565+0000 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.858+0000 7f896843e640 1 Processor -- start 2024-03-25T19:20:18.863 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.858+0000 7f896843e640 1 -- start start 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.858+0000 7f896843e640 10 monclient: init 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 10 monclient(hunting): picked mon.noname-b con 0x7f89600b4700 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 10 monclient(hunting): picked mon.noname-c con 0x7f8960142a20 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 10 monclient(hunting): picked mon.noname-a con 0x7f89600a96b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600a9d60 con 0x7f89600a96b0 2024-03-25T19:20:18.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896013ff10 con 0x7f8960142a20 2024-03-25T19:20:18.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600aa1f0 con 0x7f89600b4700 2024-03-25T19:20:18.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:18.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.859+0000 7f896843e640 10 monclient(hunting): authenticate will time out at 6327.133603s 2024-03-25T19:20:18.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.860+0000 7f89659d9640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8960142a20 0x7f896013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48126/0 (socket says 172.21.15.90:48126) 2024-03-25T19:20:18.865 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.860+0000 7f89659d9640 1 -- 172.21.15.90:0/1130597050 learned_addr learned my addr 172.21.15.90:0/1130597050 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:18.866 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.862+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48080186 0 0) 0x7f896013ff10 con 0x7f8960142a20 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.862+0000 7f89651d8640 10 monclient(hunting): _negotiate 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.862+0000 7f89651d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.862+0000 7f89651d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.862+0000 7f89651d8640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.862+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f894c003520 con 0x7f8960142a20 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329076740 0 0) 0x7f89600aa1f0 con 0x7f89600b4700 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): _negotiate 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:18.867 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f896013ff10 con 0x7f89600b4700 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657851729 0 0) 0x7f89600a9d60 con 0x7f89600a96b0 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): _negotiate 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:20:18.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.863+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89600aa1f0 con 0x7f89600a96b0 2024-03-25T19:20:18.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183010089 0 0) 0x7f89600aa1f0 con 0x7f89600a96b0 2024-03-25T19:20:18.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 >> v1:172.21.15.60:6790/0 conn(0x7f89600b4700 legacy=0x7f8960142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 >> v1:172.21.15.90:6789/0 conn(0x7f8960142a20 legacy=0x7f896013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 10 monclient: _finish_hunting 0 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 1 monclient: found mon.noname-a 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89600b4ae0 con 0x7f89600a96b0 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 10 monclient: _finish_auth 0 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 10 monclient: _check_auth_tickets 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f896843e640 5 monclient: authenticate success, global_id 4116 2024-03-25T19:20:18.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8958001930 con 0x7f89600a96b0 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f896843e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f89651d8640 10 monclient: dump: 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:18.874 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.867+0000 7f896843e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8958001a90 con 0x7f89600a96b0 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f89651d8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f896843e640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f89651d8640 1 -- 172.21.15.90:0/1130597050 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8958001ee0 con 0x7f89600a96b0 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f896843e640 4 set_mon_vals no callback set 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f89651d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f89651d8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:18.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f89651d8640 10 monclient: dump: 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f896843e640 10 monclient: shutdown 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f896843e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f896843e640 1 -- 172.21.15.90:0/1130597050 >> v1:172.21.15.60:6789/0 conn(0x7f89600a96b0 legacy=0x7f896013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f896843e640 1 -- 172.21.15.90:0/1130597050 shutdown_connections 2024-03-25T19:20:18.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.869+0000 7f896843e640 1 -- 172.21.15.90:0/1130597050 >> 172.21.15.90:0/1130597050 conn(0x7f89600668e0 msgr2=0x7f8960066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.870+0000 7f896843e640 1 -- 172.21.15.90:0/1130597050 shutdown_connections 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.870+0000 7f896843e640 1 -- 172.21.15.90:0/1130597050 wait complete. 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.870+0000 7f896843e640 10 monclient: build_initial_monmap 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.870+0000 7f896843e640 10 monclient: monmap: 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:18.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.870+0000 7f896843e640 1 librados: starting msgr at 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.870+0000 7f896843e640 1 librados: starting objecter 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.870+0000 7f896843e640 1 Processor -- start 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 1 -- start start 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 1 librados: setting wanted keys 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 1 librados: calling monclient init 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient: init 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient(hunting): picked mon.noname-b con 0x7f89600a96b0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient(hunting): picked mon.noname-c con 0x7f89600b4700 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient(hunting): picked mon.noname-a con 0x7f8960142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600b1850 con 0x7f8960142a20 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600ada00 con 0x7f89600a96b0 2024-03-25T19:20:18.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600adc70 con 0x7f89600b4700 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.871+0000 7f896843e640 10 monclient(hunting): authenticate will time out at 6327.145429s 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.872+0000 7f89659d9640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f89600b4700 0x7f89600ac400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.90:34210/0 (socket says 172.21.15.90:34210) 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.872+0000 7f89659d9640 1 -- 172.21.15.90:0/3087128416 learned_addr learned my addr 172.21.15.90:0/3087128416 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.872+0000 7f8956ffd640 10 client.?.objecter ms_handle_connect 0x7f89600b4700 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.872+0000 7f8956ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.872+0000 7f8956ffd640 10 client.?.objecter ms_handle_connect 0x7f8960142a20 2024-03-25T19:20:18.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.872+0000 7f8956ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:18.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277245346 0 0) 0x7f89600adc70 con 0x7f89600b4700 2024-03-25T19:20:18.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:18.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:18.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:18.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:20:18.880 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8944004eb0 con 0x7f89600b4700 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469782117 0 0) 0x7f89600b1850 con 0x7f8960142a20 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.874+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89600adc70 con 0x7f8960142a20 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.879+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858167188 0 0) 0x7f89600adc70 con 0x7f8960142a20 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.879+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89600b1850 con 0x7f8960142a20 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.879+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8958001da0 con 0x7f8960142a20 2024-03-25T19:20:18.881 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.879+0000 7f8956ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.879+0000 7f8956ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.879+0000 7f8956ffd640 10 monclient(hunting): dump: 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:18.882 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:18.883 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:18.883 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.879+0000 7f8956ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 799395784 0 0) 0x7f89600b1850 con 0x7f8960142a20 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 >> v1:172.21.15.60:6790/0 conn(0x7f89600b4700 legacy=0x7f89600ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 >> v1:172.21.15.90:6789/0 conn(0x7f89600a96b0 legacy=0x7f896013b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 1 monclient: found mon.a 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:18.885 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89600adf50 con 0x7f8960142a20 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 10 monclient: _finish_auth 0 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f8956ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 10 monclient: _renew_subs 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89600acc40 con 0x7f8960142a20 2024-03-25T19:20:18.886 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 10 monclient: _renew_subs 2024-03-25T19:20:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89601e32a0 con 0x7f8960142a20 2024-03-25T19:20:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.883+0000 7f896843e640 1 librados: init done 2024-03-25T19:20:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.884+0000 7f896843e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.884+0000 7f896843e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:18.887 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.884+0000 7f896843e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:18.889 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.884+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896004f730 con 0x7f8960142a20 2024-03-25T19:20:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8958002020 con 0x7f8960142a20 2024-03-25T19:20:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8958003dc0 con 0x7f8960142a20 2024-03-25T19:20:18.890 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 10 monclient: dump: 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:18.891 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f89580042b0 con 0x7f8960142a20 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8957fff640 4 set_mon_vals no callback set 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1395339450 0 0) 0x7f89580055b0 con 0x7f8960142a20 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 10 client.4119.objecter ms_dispatch 0x7f8960000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:20:18.892 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:20:18.893 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.888+0000 7f8956ffd640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:20:18.893 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:18.889+0000 7faf1d632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:20:18.945 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:18.945+0000 7faf1d632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:20:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.987+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8958005990 con 0x7f8960142a20 2024-03-25T19:20:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.987+0000 7f8956ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:18.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:18.987+0000 7f8956ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:18.990 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:18.990+0000 7faf1d632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:20:19.045 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.044+0000 7faf1d632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:20:19.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.057+0000 7f896843e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "63dcc61d-0867-4284-a895-94136baa98f9", "id": 0}] 2024-03-25T19:20:19.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.057+0000 7f896843e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "63dcc61d-0867-4284-a895-94136baa98f9", "id": 0}] 2024-03-25T19:20:19.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.057+0000 7f896843e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.058 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.057+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd new", "uuid": "63dcc61d-0867-4284-a895-94136baa98f9", "id": 0} v 0) v1 -- 0x7f89600b74a0 con 0x7f8960142a20 2024-03-25T19:20:19.086 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.086+0000 7faf1d632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:20:19.168 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.168+0000 7f8956ffd640 1 -- 172.21.15.90:0/3087128416 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "63dcc61d-0867-4284-a895-94136baa98f9", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 4236841203 0 732707515) 0x7f8958004560 con 0x7f8960142a20 2024-03-25T19:20:19.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.168+0000 7f8956ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "63dcc61d-0867-4284-a895-94136baa98f9", "id": 0}] 2024-03-25T19:20:19.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.168+0000 7f8956ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:19.169 INFO:teuthology.orchestra.run.smithi090.stdout:0 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 10 librados: watch_flush enter 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 10 librados: watch_flush exit 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 10 monclient: shutdown 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 >> v1:172.21.15.60:6789/0 conn(0x7f8960142a20 legacy=0x7f89600ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.170 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 shutdown_connections 2024-03-25T19:20:19.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 >> 172.21.15.90:0/3087128416 conn(0x7f89600668e0 msgr2=0x7f89600b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:19.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 shutdown_connections 2024-03-25T19:20:19.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 1 -- 172.21.15.90:0/3087128416 wait complete. 2024-03-25T19:20:19.171 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.169+0000 7f896843e640 1 librados: shutdown 2024-03-25T19:20:19.179 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.179+0000 7fd81362b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:20:19.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 14a85c8d-cc53-4f18-8189-46f512c990db 1 2024-03-25T19:20:19.206 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.206+0000 7faf1d632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:20:19.237 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.237+0000 7fd81362b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:20:19.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.280+0000 7f8fdd9a8640 10 monclient: get_monmap_and_config 2024-03-25T19:20:19.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.280+0000 7f8fdd9a8640 10 monclient: build_initial_monmap 2024-03-25T19:20:19.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.280+0000 7f8fdd9a8640 10 monclient: monmap: 2024-03-25T19:20:19.280 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:19.281933+0000 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:19.281933+0000 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:19.281 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.281 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.281+0000 7faf1d632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:20:19.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 1 Processor -- start 2024-03-25T19:20:19.282 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 1 -- start start 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient: init 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd80a9920 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd80b45a0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd8146450 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd806b100 con 0x7f8fd80b45a0 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80b2940 con 0x7f8fd80a9920 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd813fec0 con 0x7f8fd8146450 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.281+0000 7f8fdd9a8640 10 monclient(hunting): authenticate will time out at 6327.555744s 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.282+0000 7f8fd6ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8fd80a9920 0x7f8fd813f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48138/0 (socket says 172.21.15.90:48138) 2024-03-25T19:20:19.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.282+0000 7f8fd6ffd640 1 -- 172.21.15.90:0/2726343991 learned_addr learned my addr 172.21.15.90:0/2726343991 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:19.284 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.284+0000 7fd81362b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.284+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635824907 0 0) 0x7f8fd80b2940 con 0x7f8fd80a9920 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.284+0000 7f8fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.284+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.284+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.284+0000 7f8fd5ffb640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.284+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc0003520 con 0x7f8fd80a9920 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551640632 0 0) 0x7f8fd813fec0 con 0x7f8fd8146450 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:20:19.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd80b2940 con 0x7f8fd8146450 2024-03-25T19:20:19.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563816885 0 0) 0x7f8fd806b100 con 0x7f8fd80b45a0 2024-03-25T19:20:19.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:20:19.286 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.285+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd813fec0 con 0x7f8fd80b45a0 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110176562 0 0) 0x7f8fd80b2940 con 0x7f8fd8146450 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 >> v1:172.21.15.90:6789/0 conn(0x7f8fd80a9920 legacy=0x7f8fd813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 >> v1:172.21.15.60:6789/0 conn(0x7f8fd80b45a0 legacy=0x7f8fd8145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 1 monclient: found mon.noname-b 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd8140050 con 0x7f8fd8146450 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8fd00019f0 con 0x7f8fd8146450 2024-03-25T19:20:19.289 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fdd9a8640 5 monclient: authenticate success, global_id 4121 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fdd9a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fd5ffb640 10 monclient: dump: 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.290 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.288+0000 7f8fdd9a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fd0001b50 con 0x7f8fd8146450 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fd5ffb640 1 -- 172.21.15.90:0/2726343991 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8fd0001dc0 con 0x7f8fd8146450 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 4 set_mon_vals no callback set 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:19.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fd5ffb640 10 monclient: dump: 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.316 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 10 monclient: shutdown 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/2726343991 >> v1:172.21.15.60:6790/0 conn(0x7f8fd8146450 legacy=0x7f8fd814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/2726343991 shutdown_connections 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/2726343991 >> 172.21.15.90:0/2726343991 conn(0x7f8fd80b7f40 msgr2=0x7f8fd80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/2726343991 shutdown_connections 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.315+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/2726343991 wait complete. 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.316+0000 7f8fdd9a8640 10 monclient: build_initial_monmap 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.316+0000 7f8fdd9a8640 10 monclient: monmap: 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:19.319 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.316+0000 7f8fdd9a8640 1 librados: starting msgr at 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.316+0000 7f8fdd9a8640 1 librados: starting objecter 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.316+0000 7f8fdd9a8640 1 Processor -- start 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 1 -- start start 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 1 librados: setting wanted keys 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 1 librados: calling monclient init 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient: init 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:19.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd80a9920 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd80b45a0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd8146450 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd80ac8a0 con 0x7f8fd80a9920 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd81da9a0 con 0x7f8fd80b45a0 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd8145430 con 0x7f8fd8146450 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.317+0000 7f8fdd9a8640 10 monclient(hunting): authenticate will time out at 6327.591543s 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fd6ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8fd80a9920 0x7f8fd8144d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39660/0 (socket says 172.21.15.90:39660) 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fd6ffd640 1 -- 172.21.15.90:0/3150856314 learned_addr learned my addr 172.21.15.90:0/3150856314 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fbffff640 10 client.?.objecter ms_handle_connect 0x7f8fd80a9920 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fbffff640 10 client.?.objecter ms_handle_connect 0x7f8fd80b45a0 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fbffff640 10 client.?.objecter ms_handle_connect 0x7f8fd8146450 2024-03-25T19:20:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.318+0000 7f8fbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870863879 0 0) 0x7f8fd80ac8a0 con 0x7f8fd80a9920 2024-03-25T19:20:19.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc4004eb0 con 0x7f8fd80a9920 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568316312 0 0) 0x7f8fd81da9a0 con 0x7f8fd80b45a0 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.320+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd80ac8a0 con 0x7f8fd80b45a0 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.321+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720475300 0 0) 0x7f8fd8145430 con 0x7f8fd8146450 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.321+0000 7f8fbffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.321+0000 7f8fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.321+0000 7f8fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.323 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.321+0000 7f8fbffff640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:20:19.324 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.321+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd81da9a0 con 0x7f8fd8146450 2024-03-25T19:20:19.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.325+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3081766040 0 0) 0x7f8fc4004eb0 con 0x7f8fd80a9920 2024-03-25T19:20:19.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.325+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fd8145430 con 0x7f8fd80a9920 2024-03-25T19:20:19.325 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.325+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8fc8001950 con 0x7f8fd80a9920 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.325+0000 7f8fbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.325+0000 7f8fbffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.325+0000 7f8fbffff640 10 monclient(hunting): dump: 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.325+0000 7f8fbffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.326+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 643034586 0 0) 0x7f8fd81da9a0 con 0x7f8fd8146450 2024-03-25T19:20:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.326+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fc4004eb0 con 0x7f8fd8146450 2024-03-25T19:20:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.326+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8fd0001e60 con 0x7f8fd8146450 2024-03-25T19:20:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.326+0000 7f8fbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.326+0000 7f8fbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:19.327 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.326+0000 7f8fbffff640 10 monclient(hunting): dump: 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.328 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 659857367 0 0) 0x7f8fd8145430 con 0x7f8fd80a9920 2024-03-25T19:20:19.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 >> v1:172.21.15.60:6790/0 conn(0x7f8fd8146450 legacy=0x7f8fd8137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 >> v1:172.21.15.90:6789/0 conn(0x7f8fd80b45a0 legacy=0x7f8fd81daec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:19.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 1 monclient: found mon.a 2024-03-25T19:20:19.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:19.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd81db5d0 con 0x7f8fd80a9920 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 10 monclient: _finish_auth 0 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 5 monclient: authenticate success, global_id 4125 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 10 monclient: _renew_subs 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd81415e0 con 0x7f8fd80a9920 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 10 monclient: _renew_subs 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd8137c00 con 0x7f8fd80a9920 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.329+0000 7f8fdd9a8640 1 librados: init done 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.330+0000 7f8fdd9a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.330+0000 7f8fdd9a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.330+0000 7f8fdd9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.330+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd81dab30 con 0x7f8fd80a9920 2024-03-25T19:20:19.332 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.331+0000 7faf1d632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:20:19.345 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.344+0000 7fd81362b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:20:19.384 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.384+0000 7faf1d632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:20:19.389 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.388+0000 7fd81362b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:20:19.431 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.431+0000 7faf1d632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fc80047e0 con 0x7f8fd80a9920 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8fc8001d80 con 0x7f8fd80a9920 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fbffff640 10 monclient: dump: 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.435 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8fc8004fc0 con 0x7f8fd80a9920 2024-03-25T19:20:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.433+0000 7f8fd57fa640 4 set_mon_vals no callback set 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 445236884 0 0) 0x7f8fc8006520 con 0x7f8fd80a9920 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 10 client.4125.objecter ms_dispatch 0x7f8fd8000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8fc80054e0 con 0x7f8fd80a9920 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.436 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.434+0000 7f8fbffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:19.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.507+0000 7f8fdd9a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "14a85c8d-cc53-4f18-8189-46f512c990db", "id": 1}] 2024-03-25T19:20:19.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.507+0000 7f8fdd9a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "14a85c8d-cc53-4f18-8189-46f512c990db", "id": 1}] 2024-03-25T19:20:19.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.507+0000 7f8fdd9a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.507 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.507+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd new", "uuid": "14a85c8d-cc53-4f18-8189-46f512c990db", "id": 1} v 0) v1 -- 0x7f8fd804f730 con 0x7f8fd80a9920 2024-03-25T19:20:19.518 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.517+0000 7fd81362b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:20:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.566+0000 7f8fbffff640 1 -- 172.21.15.90:0/3150856314 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "14a85c8d-cc53-4f18-8189-46f512c990db", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4213323751 0 940483276) 0x7f8fc80058d0 con 0x7f8fd80a9920 2024-03-25T19:20:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.566+0000 7f8fbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "14a85c8d-cc53-4f18-8189-46f512c990db", "id": 1}] 2024-03-25T19:20:19.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.566+0000 7f8fbffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 10 librados: watch_flush enter 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 10 librados: watch_flush exit 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 10 client.4125.objecter successfully canceled tick 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 10 monclient: shutdown 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 >> v1:172.21.15.60:6789/0 conn(0x7f8fd80a9920 legacy=0x7f8fd8144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 shutdown_connections 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 >> 172.21.15.90:0/3150856314 conn(0x7f8fd80b7f40 msgr2=0x7f8fd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 shutdown_connections 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 1 -- 172.21.15.90:0/3150856314 wait complete. 2024-03-25T19:20:19.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.567+0000 7f8fdd9a8640 1 librados: shutdown 2024-03-25T19:20:19.577 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new d670f672-5f2a-46b2-a7d5-d2957956e0c4 2 2024-03-25T19:20:19.602 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.602+0000 7fd81362b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:20:19.650 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.650+0000 7fd81362b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:20:19.708 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.708+0000 7fd81362b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.719+0000 7f62ea396640 10 monclient: get_monmap_and_config 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.719+0000 7f62ea396640 10 monclient: build_initial_monmap 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.719+0000 7f62ea396640 10 monclient: monmap: 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:19.720306+0000 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:19.720306+0000 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.719 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:19.720 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:19.720 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:19.720 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.719+0000 7f62ea396640 1 Processor -- start 2024-03-25T19:20:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.719+0000 7f62ea396640 1 -- start start 2024-03-25T19:20:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.719+0000 7f62ea396640 10 monclient: init 2024-03-25T19:20:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:19.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 10 monclient(hunting): picked mon.noname-c con 0x7f62e40b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 10 monclient(hunting): picked mon.noname-a con 0x7f62e4142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 10 monclient(hunting): picked mon.noname-b con 0x7f62e40a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e40a9d60 con 0x7f62e4142a20 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e413ff10 con 0x7f62e40b4700 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e40aa1f0 con 0x7f62e40a96b0 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62ea396640 10 monclient(hunting): authenticate will time out at 6327.994334s 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62e37fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f62e4142a20 0x7f62e413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39662/0 (socket says 172.21.15.90:39662) 2024-03-25T19:20:19.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.720+0000 7f62e37fe640 1 -- 172.21.15.90:0/4183640502 learned_addr learned my addr 172.21.15.90:0/4183640502 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186474039 0 0) 0x7f62e40a9d60 con 0x7f62e4142a20 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc003520 con 0x7f62e4142a20 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853057896 0 0) 0x7f62e413ff10 con 0x7f62e40b4700 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:20:19.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e40a9d60 con 0x7f62e40b4700 2024-03-25T19:20:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154474096 0 0) 0x7f62e40aa1f0 con 0x7f62e40a96b0 2024-03-25T19:20:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:20:19.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.723+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e413ff10 con 0x7f62e40a96b0 2024-03-25T19:20:19.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1483089241 0 0) 0x7f62e40a9d60 con 0x7f62e40b4700 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 >> v1:172.21.15.60:6790/0 conn(0x7f62e40a96b0 legacy=0x7f62e413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 >> v1:172.21.15.60:6789/0 conn(0x7f62e4142a20 legacy=0x7f62e413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e40b4ae0 con 0x7f62e40b4700 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f62d4001670 con 0x7f62e40b4700 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62e2ffd640 10 monclient: dump: 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.727 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62ea396640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:20:19.728 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.726+0000 7f62ea396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:19.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d4001850 con 0x7f62e40b4700 2024-03-25T19:20:19.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:19.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62e2ffd640 1 -- 172.21.15.90:0/4183640502 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f62d4001c60 con 0x7f62e40b4700 2024-03-25T19:20:19.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:19.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62e2ffd640 10 monclient: dump: 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62ea396640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:19.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.752+0000 7f62ea396640 4 set_mon_vals no callback set 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.753+0000 7f62ea396640 10 monclient: shutdown 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.753+0000 7f62ea396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.753+0000 7f62ea396640 1 -- 172.21.15.90:0/4183640502 >> v1:172.21.15.90:6789/0 conn(0x7f62e40b4700 legacy=0x7f62e4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.753+0000 7f62ea396640 1 -- 172.21.15.90:0/4183640502 shutdown_connections 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.753+0000 7f62ea396640 1 -- 172.21.15.90:0/4183640502 >> 172.21.15.90:0/4183640502 conn(0x7f62e40668e0 msgr2=0x7f62e4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.753+0000 7f62ea396640 1 -- 172.21.15.90:0/4183640502 shutdown_connections 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.753+0000 7f62ea396640 1 -- 172.21.15.90:0/4183640502 wait complete. 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 10 monclient: build_initial_monmap 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 10 monclient: monmap: 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:19.756 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 1 librados: starting msgr at 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 1 librados: starting objecter 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 1 Processor -- start 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 1 -- start start 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 1 librados: setting wanted keys 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 1 librados: calling monclient init 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.754+0000 7f62ea396640 10 monclient: init 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:19.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 10 monclient(hunting): picked mon.noname-a con 0x7f62e40a96b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 10 monclient(hunting): picked mon.noname-b con 0x7f62e40b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 10 monclient(hunting): picked mon.noname-c con 0x7f62e4142a20 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e412d9a0 con 0x7f62e40a96b0 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e4137700 con 0x7f62e40b4700 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e41379e0 con 0x7f62e4142a20 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62ea396640 10 monclient(hunting): authenticate will time out at 6328.029132s 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62e37fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f62e40b4700 0x7f62e412cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48150/0 (socket says 172.21.15.90:48150) 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62e37fe640 1 -- 172.21.15.90:0/847475832 learned_addr learned my addr 172.21.15.90:0/847475832 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62e0ff9640 10 client.?.objecter ms_handle_connect 0x7f62e40b4700 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.755+0000 7f62e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.756+0000 7f62e0ff9640 10 client.?.objecter ms_handle_connect 0x7f62e40a96b0 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.756+0000 7f62e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.756+0000 7f62e0ff9640 10 client.?.objecter ms_handle_connect 0x7f62e4142a20 2024-03-25T19:20:19.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.756+0000 7f62e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.757+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829543472 0 0) 0x7f62e4137700 con 0x7f62e40b4700 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.757+0000 7f62e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.757+0000 7f62e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.757+0000 7f62e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.757+0000 7f62e0ff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.757+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d0004eb0 con 0x7f62e40b4700 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851898988 0 0) 0x7f62e412d9a0 con 0x7f62e40a96b0 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e4137700 con 0x7f62e40a96b0 2024-03-25T19:20:19.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266158847 0 0) 0x7f62e41379e0 con 0x7f62e4142a20 2024-03-25T19:20:19.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:19.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:19.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:19.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:20:19.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.758+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62e412d9a0 con 0x7f62e4142a20 2024-03-25T19:20:19.761 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:19.760+0000 7fd81362b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996357149 0 0) 0x7f62d0004eb0 con 0x7f62e40b4700 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62e41379e0 con 0x7f62e40b4700 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f62d8001950 con 0x7f62e40b4700 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.763 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2156983554 0 0) 0x7f62e4137700 con 0x7f62e40a96b0 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.762+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62d0004eb0 con 0x7f62e40a96b0 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.763+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f62d4001ba0 con 0x7f62e40a96b0 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.763+0000 7f62e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.763+0000 7f62e0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.763+0000 7f62e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.764 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.764+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 97920209 0 0) 0x7f62e412d9a0 con 0x7f62e4142a20 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.764+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62e4137700 con 0x7f62e4142a20 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.764+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f62dc001bd0 con 0x7f62e4142a20 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.764+0000 7f62e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.764+0000 7f62e0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.764+0000 7f62e0ff9640 10 monclient(hunting): dump: 2024-03-25T19:20:19.765 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.766 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3325093281 0 0) 0x7f62d0004eb0 con 0x7f62e40a96b0 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 >> v1:172.21.15.60:6790/0 conn(0x7f62e4142a20 legacy=0x7f62e412d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 >> v1:172.21.15.90:6789/0 conn(0x7f62e40b4700 legacy=0x7f62e412cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 1 monclient: found mon.a 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e4136550 con 0x7f62e40a96b0 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:20:19.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 5 monclient: authenticate success, global_id 4131 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 10 monclient: _renew_subs 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e4133780 con 0x7f62e40a96b0 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 10 monclient: _renew_subs 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e4135cb0 con 0x7f62e40a96b0 2024-03-25T19:20:19.768 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.766+0000 7f62ea396640 1 librados: init done 2024-03-25T19:20:19.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.767+0000 7f62ea396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.767+0000 7f62ea396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.767+0000 7f62ea396640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.767+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e4133a10 con 0x7f62e40a96b0 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62d4003960 con 0x7f62e40a96b0 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f62d4001230 con 0x7f62e40a96b0 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 10 monclient: dump: 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.771 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f62d4004000 con 0x7f62e40a96b0 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1596841565 0 0) 0x7f62d4005530 con 0x7f62e40a96b0 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 10 client.4131.objecter ms_dispatch 0x7f62e4000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e27fc640 4 set_mon_vals no callback set 2024-03-25T19:20:19.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.770+0000 7f62e0ff9640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T19:20:19.833 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.833+0000 7faf1d632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:20:19.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.870+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f62d40051c0 con 0x7f62e40a96b0 2024-03-25T19:20:19.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.870+0000 7f62e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:19.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.870+0000 7f62e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:19.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.943+0000 7f62ea396640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d670f672-5f2a-46b2-a7d5-d2957956e0c4", "id": 2}] 2024-03-25T19:20:19.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.943+0000 7f62ea396640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d670f672-5f2a-46b2-a7d5-d2957956e0c4", "id": 2}] 2024-03-25T19:20:19.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.943+0000 7f62ea396640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:19.944 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.943+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d670f672-5f2a-46b2-a7d5-d2957956e0c4", "id": 2} v 0) v1 -- 0x7f62e404f730 con 0x7f62e40a96b0 2024-03-25T19:20:19.970 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:19.970+0000 7faf1d632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:20:19.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.990+0000 7f62e0ff9640 1 -- 172.21.15.90:0/847475832 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d670f672-5f2a-46b2-a7d5-d2957956e0c4", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 321975473 0 216599125) 0x7f62d4005740 con 0x7f62e40a96b0 2024-03-25T19:20:19.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.990+0000 7f62e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d670f672-5f2a-46b2-a7d5-d2957956e0c4", "id": 2}] 2024-03-25T19:20:19.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.990+0000 7f62e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:19.991 INFO:teuthology.orchestra.run.smithi090.stdout:2 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 10 librados: watch_flush enter 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 10 librados: watch_flush exit 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 10 client.4131.objecter successfully canceled tick 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 10 monclient: shutdown 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 >> v1:172.21.15.60:6789/0 conn(0x7f62e40a96b0 legacy=0x7f62e4133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 shutdown_connections 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 >> 172.21.15.90:0/847475832 conn(0x7f62e40668e0 msgr2=0x7f62e40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:19.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 shutdown_connections 2024-03-25T19:20:19.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 1 -- 172.21.15.90:0/847475832 wait complete. 2024-03-25T19:20:19.993 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:19.991+0000 7f62ea396640 1 librados: shutdown 2024-03-25T19:20:20.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 98f1537b-88fb-42c5-b3b6-af8d02f925f4 3 2024-03-25T19:20:20.098 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:20.098+0000 7faf1d632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:20:20.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.100+0000 7fcd17ed3640 10 monclient: get_monmap_and_config 2024-03-25T19:20:20.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.100+0000 7fcd17ed3640 10 monclient: build_initial_monmap 2024-03-25T19:20:20.100 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.100+0000 7fcd17ed3640 10 monclient: monmap: 2024-03-25T19:20:20.100 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:20.100 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.100 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:20.101732+0000 2024-03-25T19:20:20.101 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:20.101732+0000 2024-03-25T19:20:20.101 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:20.101 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.101 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:20.101 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:20.101 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:20.101 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 1 Processor -- start 2024-03-25T19:20:20.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 1 -- start start 2024-03-25T19:20:20.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient: init 2024-03-25T19:20:20.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:20.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:20.102 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient(hunting): picked mon.noname-b con 0x7fcd100b4700 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient(hunting): picked mon.noname-c con 0x7fcd10142a20 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient(hunting): picked mon.noname-a con 0x7fcd100a96b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd100a9d60 con 0x7fcd100a96b0 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1013ff10 con 0x7fcd10142a20 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd100aa1f0 con 0x7fcd100b4700 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.101+0000 7fcd17ed3640 10 monclient(hunting): authenticate will time out at 6328.375668s 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.102+0000 7fcd1546e640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcd10142a20 0x7fcd1013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48162/0 (socket says 172.21.15.90:48162) 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.102+0000 7fcd16470640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fcd100a96b0 0x7fcd1013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39684/0 (socket says 172.21.15.90:39684) 2024-03-25T19:20:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.102+0000 7fcd1546e640 1 -- 172.21.15.90:0/1459710559 learned_addr learned my addr 172.21.15.90:0/1459710559 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:20.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.104+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580125384 0 0) 0x7fcd1013ff10 con 0x7fcd10142a20 2024-03-25T19:20:20.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.104+0000 7fcd14c6d640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.104+0000 7fcd14c6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.104+0000 7fcd14c6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.104+0000 7fcd14c6d640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.104+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccfc003520 con 0x7fcd10142a20 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215787157 0 0) 0x7fcd100a9d60 con 0x7fcd100a96b0 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:20:20.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd1013ff10 con 0x7fcd100a96b0 2024-03-25T19:20:20.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320420107 0 0) 0x7fcd100aa1f0 con 0x7fcd100b4700 2024-03-25T19:20:20.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:20:20.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.105+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd100a9d60 con 0x7fcd100b4700 2024-03-25T19:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126846037 0 0) 0x7fccfc003520 con 0x7fcd10142a20 2024-03-25T19:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 >> v1:172.21.15.60:6790/0 conn(0x7fcd100b4700 legacy=0x7fcd10142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 >> v1:172.21.15.60:6789/0 conn(0x7fcd100a96b0 legacy=0x7fcd1013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 10 monclient: _finish_hunting 0 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 1 monclient: found mon.noname-c 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd100b4ae0 con 0x7fcd10142a20 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 10 monclient: _finish_auth 0 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 10 monclient: _check_auth_tickets 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fcd00001690 con 0x7fcd10142a20 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd17ed3640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd17ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd14c6d640 10 monclient: dump: 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.109 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.107+0000 7fcd17ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd000017f0 con 0x7fcd10142a20 2024-03-25T19:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd14c6d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd14c6d640 1 -- 172.21.15.90:0/1459710559 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fcd00001c00 con 0x7fcd10142a20 2024-03-25T19:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 4 set_mon_vals no callback set 2024-03-25T19:20:20.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd14c6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd14c6d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd14c6d640 10 monclient: dump: 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.112 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 10 monclient: shutdown 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 1 -- 172.21.15.90:0/1459710559 >> v1:172.21.15.90:6789/0 conn(0x7fcd10142a20 legacy=0x7fcd1013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 1 -- 172.21.15.90:0/1459710559 shutdown_connections 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 1 -- 172.21.15.90:0/1459710559 >> 172.21.15.90:0/1459710559 conn(0x7fcd100668e0 msgr2=0x7fcd10066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 1 -- 172.21.15.90:0/1459710559 shutdown_connections 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.110+0000 7fcd17ed3640 1 -- 172.21.15.90:0/1459710559 wait complete. 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.111+0000 7fcd17ed3640 10 monclient: build_initial_monmap 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.111+0000 7fcd17ed3640 10 monclient: monmap: 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:20.113 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:20.114 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 librados: starting msgr at 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 librados: starting objecter 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 Processor -- start 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 -- start start 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 librados: setting wanted keys 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 librados: calling monclient init 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient: init 2024-03-25T19:20:20.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient(hunting): picked mon.noname-a con 0x7fcd100a96b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient(hunting): picked mon.noname-b con 0x7fcd100b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient(hunting): picked mon.noname-c con 0x7fcd10142a20 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd101d7590 con 0x7fcd100a96b0 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1013ccc0 con 0x7fcd100b4700 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd101d8180 con 0x7fcd10142a20 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.112+0000 7fcd17ed3640 10 monclient(hunting): authenticate will time out at 6328.386589s 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.113+0000 7fcd1546e640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fcd100b4700 0x7fcd101d4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48174/0 (socket says 172.21.15.90:48174) 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.113+0000 7fcd1546e640 1 -- 172.21.15.90:0/3578164206 learned_addr learned my addr 172.21.15.90:0/3578164206 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.113+0000 7fcd067fc640 10 client.?.objecter ms_handle_connect 0x7fcd100b4700 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.113+0000 7fcd067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.113+0000 7fcd067fc640 10 client.?.objecter ms_handle_connect 0x7fcd10142a20 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.113+0000 7fcd067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:20.117 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651303874 0 0) 0x7fcd1013ccc0 con 0x7fcd100b4700 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf4004eb0 con 0x7fcd100b4700 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552700668 0 0) 0x7fcd101d8180 con 0x7fcd10142a20 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd1013ccc0 con 0x7fcd10142a20 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 client.?.objecter ms_handle_connect 0x7fcd100a96b0 2024-03-25T19:20:20.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.115+0000 7fcd067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:20.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.118+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442890992 0 0) 0x7fcd101d7590 con 0x7fcd100a96b0 2024-03-25T19:20:20.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.118+0000 7fcd067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.118+0000 7fcd067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.118+0000 7fcd067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.118+0000 7fcd067fc640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:20:20.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.118+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd101d8180 con 0x7fcd100a96b0 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.121+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012856776 0 0) 0x7fccf4004eb0 con 0x7fcd100b4700 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.121+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd101d7590 con 0x7fcd100b4700 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.121+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fcd00001bc0 con 0x7fcd100b4700 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.121+0000 7fcd067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.121+0000 7fcd067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.121+0000 7fcd067fc640 10 monclient(hunting): dump: 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.122 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.121+0000 7fcd067fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642636453 0 0) 0x7fcd1013ccc0 con 0x7fcd10142a20 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccf4004eb0 con 0x7fcd10142a20 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fcd08001bd0 con 0x7fcd10142a20 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 10 monclient(hunting): dump: 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.124 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371235532 0 0) 0x7fcd101d8180 con 0x7fcd100a96b0 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd1013ccc0 con 0x7fcd100a96b0 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fcd0c001830 con 0x7fcd100a96b0 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.123+0000 7fcd067fc640 10 monclient(hunting): dump: 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.125 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.126 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.126 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4254075325 0 0) 0x7fcd101d7590 con 0x7fcd100b4700 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 >> v1:172.21.15.60:6790/0 conn(0x7fcd10142a20 legacy=0x7fcd1013b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 >> v1:172.21.15.60:6789/0 conn(0x7fcd100a96b0 legacy=0x7fcd101d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 1 monclient: found mon.b 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.124+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd101d8310 con 0x7fcd100b4700 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd067fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:20.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 5 monclient: authenticate success, global_id 4132 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 10 monclient: _renew_subs 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd101d3fd0 con 0x7fcd100b4700 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 10 monclient: _renew_subs 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd101d4f40 con 0x7fcd100b4700 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.125+0000 7fcd17ed3640 1 librados: init done 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.126+0000 7fcd17ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.126+0000 7fcd17ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.126+0000 7fcd17ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:20.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.126+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd101d73f0 con 0x7fcd100b4700 2024-03-25T19:20:20.148 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:20.149+0000 7faf1d632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:20:20.162 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.162+0000 7fd81362b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:20:20.265 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:20.265+0000 7faf1d632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:20:20.302 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.302+0000 7fd81362b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd000011f0 con 0x7fcd100b4700 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fcd00004b10 con 0x7fcd100b4700 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 10 monclient: dump: 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.321 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcd00005080 con 0x7fcd100b4700 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd07fff640 4 set_mon_vals no callback set 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2863594263 0 0) 0x7fcd000066f0 con 0x7fcd100b4700 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 10 client.4132.objecter ms_dispatch 0x7fcd10000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcd000069b0 con 0x7fcd100b4700 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:20.322 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.320+0000 7fcd067fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:20.364 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:20.364+0000 7faf1d632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:20:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.394+0000 7fcd17ed3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "98f1537b-88fb-42c5-b3b6-af8d02f925f4", "id": 3}] 2024-03-25T19:20:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.394+0000 7fcd17ed3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "98f1537b-88fb-42c5-b3b6-af8d02f925f4", "id": 3}] 2024-03-25T19:20:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.394+0000 7fcd17ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:20.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.394+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd new", "uuid": "98f1537b-88fb-42c5-b3b6-af8d02f925f4", "id": 3} v 0) v1 -- 0x7fcd1004f730 con 0x7fcd100b4700 2024-03-25T19:20:20.432 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.431+0000 7fd81362b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:20:20.483 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.483+0000 7fd81362b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:20:20.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.500+0000 7fcd067fc640 1 -- 172.21.15.90:0/3578164206 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98f1537b-88fb-42c5-b3b6-af8d02f925f4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3415566759 0 525046306) 0x7fcd00006390 con 0x7fcd100b4700 2024-03-25T19:20:20.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.500+0000 7fcd067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "98f1537b-88fb-42c5-b3b6-af8d02f925f4", "id": 3}] 2024-03-25T19:20:20.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.500+0000 7fcd067fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:20.501 INFO:teuthology.orchestra.run.smithi090.stdout:3 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 10 librados: watch_flush enter 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 10 librados: watch_flush exit 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 10 client.4132.objecter successfully canceled tick 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 10 monclient: shutdown 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 >> v1:172.21.15.90:6789/0 conn(0x7fcd100b4700 legacy=0x7fcd101d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 shutdown_connections 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 >> 172.21.15.90:0/3578164206 conn(0x7fcd100668e0 msgr2=0x7fcd100b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 shutdown_connections 2024-03-25T19:20:20.502 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 1 -- 172.21.15.90:0/3578164206 wait complete. 2024-03-25T19:20:20.503 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.501+0000 7fcd17ed3640 1 librados: shutdown 2024-03-25T19:20:20.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 5444b0c8-c71c-4fa2-b901-c4660901c646 4 2024-03-25T19:20:20.531 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:20.531+0000 7faf1d632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:20:20.576 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:20.575+0000 7faf1d632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:20:20.604 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.603+0000 7fd81362b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.612+0000 7fa556857640 10 monclient: get_monmap_and_config 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.612+0000 7fa556857640 10 monclient: build_initial_monmap 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.612+0000 7fa556857640 10 monclient: monmap: 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:20.613331+0000 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:20.613331+0000 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:20.612 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.613 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:20.613 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:20.613 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:20.613 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.612+0000 7fa556857640 1 Processor -- start 2024-03-25T19:20:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.612+0000 7fa556857640 1 -- start start 2024-03-25T19:20:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.612+0000 7fa556857640 10 monclient: init 2024-03-25T19:20:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:20.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 10 monclient(hunting): picked mon.noname-c con 0x7fa5500b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 10 monclient(hunting): picked mon.noname-b con 0x7fa550142a20 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 10 monclient(hunting): picked mon.noname-a con 0x7fa5500a96b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5500a9d60 con 0x7fa5500a96b0 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55013ff10 con 0x7fa5500b4700 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5500aa1f0 con 0x7fa550142a20 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.613+0000 7fa556857640 10 monclient(hunting): authenticate will time out at 6328.887340s 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.614+0000 7fa554df4640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa5500a96b0 0x7fa55013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39696/0 (socket says 172.21.15.90:39696) 2024-03-25T19:20:20.615 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.614+0000 7fa554df4640 1 -- 172.21.15.90:0/1657190005 learned_addr learned my addr 172.21.15.90:0/1657190005 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:20.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.616+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824851121 0 0) 0x7fa5500aa1f0 con 0x7fa550142a20 2024-03-25T19:20:20.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.616+0000 7fa54effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.616+0000 7fa54effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.616+0000 7fa54effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.616+0000 7fa54effd640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:20:20.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.616+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa538003520 con 0x7fa550142a20 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322921434 0 0) 0x7fa55013ff10 con 0x7fa5500b4700 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5500aa1f0 con 0x7fa5500b4700 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323695836 0 0) 0x7fa5500a9d60 con 0x7fa5500a96b0 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:20:20.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.617+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55013ff10 con 0x7fa5500a96b0 2024-03-25T19:20:20.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3233733506 0 0) 0x7fa538003520 con 0x7fa550142a20 2024-03-25T19:20:20.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 >> v1:172.21.15.90:6789/0 conn(0x7fa5500b4700 legacy=0x7fa550142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 >> v1:172.21.15.60:6789/0 conn(0x7fa5500a96b0 legacy=0x7fa55013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:20.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 1 monclient: found mon.noname-b 2024-03-25T19:20:20.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5500b4ae0 con 0x7fa550142a20 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 10 monclient: _finish_auth 0 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa544001690 con 0x7fa550142a20 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa556857640 5 monclient: authenticate success, global_id 4139 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa556857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa54effd640 10 monclient: dump: 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.621 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.622 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.620+0000 7fa556857640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5440017f0 con 0x7fa550142a20 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa54effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa54effd640 1 -- 172.21.15.90:0/1657190005 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa544001c00 con 0x7fa550142a20 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa54effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa54effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa54effd640 10 monclient: dump: 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.623 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.624 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:20.624 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 4 set_mon_vals no callback set 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 10 monclient: shutdown 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 1 -- 172.21.15.90:0/1657190005 >> v1:172.21.15.60:6790/0 conn(0x7fa550142a20 legacy=0x7fa55013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 1 -- 172.21.15.90:0/1657190005 shutdown_connections 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 1 -- 172.21.15.90:0/1657190005 >> 172.21.15.90:0/1657190005 conn(0x7fa5500668e0 msgr2=0x7fa550066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 1 -- 172.21.15.90:0/1657190005 shutdown_connections 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.623+0000 7fa556857640 1 -- 172.21.15.90:0/1657190005 wait complete. 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.624+0000 7fa556857640 10 monclient: build_initial_monmap 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.624+0000 7fa556857640 10 monclient: monmap: 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:20.626 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.624+0000 7fa556857640 1 librados: starting msgr at 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.624+0000 7fa556857640 1 librados: starting objecter 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.624+0000 7fa556857640 1 Processor -- start 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 1 -- start start 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 1 librados: setting wanted keys 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 1 librados: calling monclient init 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient: init 2024-03-25T19:20:20.627 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient(hunting): picked mon.noname-c con 0x7fa5500a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient(hunting): picked mon.noname-a con 0x7fa5500b4700 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient(hunting): picked mon.noname-b con 0x7fa550142a20 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5500ac3d0 con 0x7fa5500b4700 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5500ae3e0 con 0x7fa550142a20 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5500ae6c0 con 0x7fa5500a96b0 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.625+0000 7fa556857640 10 monclient(hunting): authenticate will time out at 6328.899471s 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.626+0000 7fa54f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa5500b4700 0x7fa550135ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39698/0 (socket says 172.21.15.90:39698) 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.626+0000 7fa54f7fe640 1 -- 172.21.15.90:0/2909314611 learned_addr learned my addr 172.21.15.90:0/2909314611 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.626+0000 7fa54cff9640 10 client.?.objecter ms_handle_connect 0x7fa5500b4700 2024-03-25T19:20:20.628 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.626+0000 7fa54cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:20.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.626+0000 7fa54cff9640 10 client.?.objecter ms_handle_connect 0x7fa5500a96b0 2024-03-25T19:20:20.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.626+0000 7fa54cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:20.629 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.627+0000 7fa54cff9640 10 client.?.objecter ms_handle_connect 0x7fa550142a20 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.627+0000 7fa54cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052056254 0 0) 0x7fa5500ac3d0 con 0x7fa5500b4700 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53c004eb0 con 0x7fa5500b4700 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236327898 0 0) 0x7fa5500ae6c0 con 0x7fa5500a96b0 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.628+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5500ac3d0 con 0x7fa5500a96b0 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.629+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333561075 0 0) 0x7fa5500ae3e0 con 0x7fa550142a20 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.629+0000 7fa54cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:20.630 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.629+0000 7fa54cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:20.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.629+0000 7fa54cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:20.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.629+0000 7fa54cff9640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:20:20.631 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.629+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5500ae6c0 con 0x7fa550142a20 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.632+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 507358185 0 0) 0x7fa53c004eb0 con 0x7fa5500b4700 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.632+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5500ae3e0 con 0x7fa5500b4700 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa544001c40 con 0x7fa5500b4700 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 10 monclient(hunting): dump: 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.633 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.634 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.634 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.634 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.634 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.634 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1786084141 0 0) 0x7fa5500ae6c0 con 0x7fa550142a20 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa53c004eb0 con 0x7fa550142a20 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa548001bd0 con 0x7fa550142a20 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.633+0000 7fa54cff9640 10 monclient(hunting): dump: 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.635 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.634+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146076561 0 0) 0x7fa5500ac3d0 con 0x7fa5500a96b0 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.634+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5500ae6c0 con 0x7fa5500a96b0 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.634+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa540001970 con 0x7fa5500a96b0 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.634+0000 7fa54cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.634+0000 7fa54cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.634+0000 7fa54cff9640 10 monclient(hunting): dump: 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.636 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.637 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.637 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.637 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.637 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.637 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.637 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.637 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 597900979 0 0) 0x7fa5500ae3e0 con 0x7fa5500b4700 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 >> v1:172.21.15.60:6790/0 conn(0x7fa5500a96b0 legacy=0x7fa5500ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 >> v1:172.21.15.90:6789/0 conn(0x7fa550142a20 legacy=0x7fa55013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 1 monclient: found mon.a 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5500abff0 con 0x7fa5500b4700 2024-03-25T19:20:20.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 10 monclient: _finish_auth 0 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa54cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 5 monclient: authenticate success, global_id 4143 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 10 monclient: _renew_subs 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5500ad510 con 0x7fa5500b4700 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 10 monclient: _renew_subs 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5501368e0 con 0x7fa5500b4700 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.636+0000 7fa556857640 1 librados: init done 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.637+0000 7fa556857640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.637+0000 7fa556857640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.637+0000 7fa556857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:20.639 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.637+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5500ac200 con 0x7fa5500b4700 2024-03-25T19:20:20.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.640+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5440011f0 con 0x7fa5500b4700 2024-03-25T19:20:20.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:20.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa544003a80 con 0x7fa5500b4700 2024-03-25T19:20:20.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:20.641 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 10 monclient: dump: 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa544004030 con 0x7fa5500b4700 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3121145351 0 0) 0x7fa5440068e0 con 0x7fa5500b4700 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 10 client.4143.objecter ms_dispatch 0x7fa550000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:20:20.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54dffb640 4 set_mon_vals no callback set 2024-03-25T19:20:20.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:20:20.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:20:20.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.641+0000 7fa54cff9640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T19:20:20.708 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.708+0000 7fd81362b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:20:20.736 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:20.735+0000 7faf1d632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:20:20.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.742+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa5440042e0 con 0x7fa5500b4700 2024-03-25T19:20:20.742 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.742+0000 7fa54cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:20.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.742+0000 7fa54cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.833+0000 7fa556857640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5444b0c8-c71c-4fa2-b901-c4660901c646", "id": 4}] 2024-03-25T19:20:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.833+0000 7fa556857640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5444b0c8-c71c-4fa2-b901-c4660901c646", "id": 4}] 2024-03-25T19:20:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.833+0000 7fa556857640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:20.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.833+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5444b0c8-c71c-4fa2-b901-c4660901c646", "id": 4} v 0) v1 -- 0x7fa55004f730 con 0x7fa5500b4700 2024-03-25T19:20:20.875 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.875+0000 7fd81362b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:20:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.876+0000 7fa54cff9640 1 -- 172.21.15.90:0/2909314611 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5444b0c8-c71c-4fa2-b901-c4660901c646", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1230263079 0 1697012583) 0x7fa544006480 con 0x7fa5500b4700 2024-03-25T19:20:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.876+0000 7fa54cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5444b0c8-c71c-4fa2-b901-c4660901c646", "id": 4}] 2024-03-25T19:20:20.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.876+0000 7fa54cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:20.877 INFO:teuthology.orchestra.run.smithi090.stdout:4 2024-03-25T19:20:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 10 librados: watch_flush enter 2024-03-25T19:20:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 10 librados: watch_flush exit 2024-03-25T19:20:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T19:20:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 10 client.4143.objecter successfully canceled tick 2024-03-25T19:20:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 10 monclient: shutdown 2024-03-25T19:20:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:20.878 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 >> v1:172.21.15.60:6789/0 conn(0x7fa5500b4700 legacy=0x7fa550135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:20.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 shutdown_connections 2024-03-25T19:20:20.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 >> 172.21.15.90:0/2909314611 conn(0x7fa5500668e0 msgr2=0x7fa550066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:20.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 shutdown_connections 2024-03-25T19:20:20.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 1 -- 172.21.15.90:0/2909314611 wait complete. 2024-03-25T19:20:20.879 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:20.877+0000 7fa556857640 1 librados: shutdown 2024-03-25T19:20:20.888 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new ae0a333a-7fb1-41ce-b567-a6322ab9b7a8 5 2024-03-25T19:20:20.922 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:20.921+0000 7fd81362b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:20:21.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.027+0000 7f960bb15640 10 monclient: get_monmap_and_config 2024-03-25T19:20:21.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.027+0000 7f960bb15640 10 monclient: build_initial_monmap 2024-03-25T19:20:21.027 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.027+0000 7f960bb15640 10 monclient: monmap: 2024-03-25T19:20:21.027 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:21.028347+0000 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:21.028347+0000 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.028 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.027+0000 7f960bb15640 1 Processor -- start 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.027+0000 7f960bb15640 1 -- start start 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.027+0000 7f960bb15640 10 monclient: init 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 10 monclient(hunting): picked mon.noname-b con 0x7f96040b4700 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 10 monclient(hunting): picked mon.noname-a con 0x7f9604142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 10 monclient(hunting): picked mon.noname-c con 0x7f96040a96b0 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96040a9d60 con 0x7f9604142a20 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960413ff10 con 0x7f96040a96b0 2024-03-25T19:20:21.030 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96040aa1f0 con 0x7f96040b4700 2024-03-25T19:20:21.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:21.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.028+0000 7f960bb15640 10 monclient(hunting): authenticate will time out at 6329.302386s 2024-03-25T19:20:21.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.029+0000 7f96090b0640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9604142a20 0x7f960413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39704/0 (socket says 172.21.15.90:39704) 2024-03-25T19:20:21.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.029+0000 7f96090b0640 1 -- 172.21.15.90:0/589630848 learned_addr learned my addr 172.21.15.90:0/589630848 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:21.031 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.029+0000 7f96098b1640 1 --1- 172.21.15.90:0/589630848 >> v1:172.21.15.60:6790/0 conn(0x7f96040b4700 0x7f9604142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.90:34298/0 (socket says 172.21.15.90:34298) 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095655442 0 0) 0x7f96040a9d60 con 0x7f9604142a20 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f0003520 con 0x7f9604142a20 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583696830 0 0) 0x7f960413ff10 con 0x7f96040a96b0 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.032 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:20:21.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.031+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96040a9d60 con 0x7f96040a96b0 2024-03-25T19:20:21.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.032+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144488404 0 0) 0x7f96040aa1f0 con 0x7f96040b4700 2024-03-25T19:20:21.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.032+0000 7f96088af640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.032+0000 7f96088af640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.032+0000 7f96088af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.032+0000 7f96088af640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:20:21.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.032+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f960413ff10 con 0x7f96040b4700 2024-03-25T19:20:21.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245556880 0 0) 0x7f96040a9d60 con 0x7f96040a96b0 2024-03-25T19:20:21.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 >> v1:172.21.15.60:6790/0 conn(0x7f96040b4700 legacy=0x7f9604142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 >> v1:172.21.15.60:6789/0 conn(0x7f9604142a20 legacy=0x7f960413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 10 monclient: _finish_hunting 0 2024-03-25T19:20:21.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 1 monclient: found mon.noname-c 2024-03-25T19:20:21.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:21.035 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96040b4ae0 con 0x7f96040a96b0 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 10 monclient: _finish_auth 0 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 10 monclient: _check_auth_tickets 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f95fc0019f0 con 0x7f96040a96b0 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f96088af640 10 monclient: dump: 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.036 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f960bb15640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:20:21.037 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.034+0000 7f960bb15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:21.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95fc001b50 con 0x7f96040a96b0 2024-03-25T19:20:21.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f96088af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:21.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f96088af640 1 -- 172.21.15.90:0/589630848 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f95fc001fa0 con 0x7f96040a96b0 2024-03-25T19:20:21.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f960bb15640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f960bb15640 4 set_mon_vals no callback set 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f96088af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f96088af640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.036+0000 7f96088af640 10 monclient: dump: 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.041 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 10 monclient: shutdown 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 1 -- 172.21.15.90:0/589630848 >> v1:172.21.15.90:6789/0 conn(0x7f96040a96b0 legacy=0x7f960413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 1 -- 172.21.15.90:0/589630848 shutdown_connections 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 1 -- 172.21.15.90:0/589630848 >> 172.21.15.90:0/589630848 conn(0x7f96040668e0 msgr2=0x7f9604066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 1 -- 172.21.15.90:0/589630848 shutdown_connections 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 1 -- 172.21.15.90:0/589630848 wait complete. 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 10 monclient: build_initial_monmap 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.037+0000 7f960bb15640 10 monclient: monmap: 2024-03-25T19:20:21.042 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 librados: starting msgr at 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 librados: starting objecter 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 Processor -- start 2024-03-25T19:20:21.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 -- start start 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 librados: setting wanted keys 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 librados: calling monclient init 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient: init 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient(hunting): picked mon.noname-c con 0x7f96040a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient(hunting): picked mon.noname-b con 0x7f96040b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient(hunting): picked mon.noname-a con 0x7f9604142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9604133a90 con 0x7f9604142a20 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9604139f50 con 0x7f96040b4700 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960413b390 con 0x7f96040a96b0 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:21.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.038+0000 7f960bb15640 10 monclient(hunting): authenticate will time out at 6329.312702s 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f960a0b2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9604142a20 0x7f9604133380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39716/0 (socket says 172.21.15.90:39716) 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f960a0b2640 1 -- 172.21.15.90:0/4274450457 learned_addr learned my addr 172.21.15.90:0/4274450457 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f95fa7fc640 10 client.?.objecter ms_handle_connect 0x7f9604142a20 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f95fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f95fa7fc640 10 client.?.objecter ms_handle_connect 0x7f96040b4700 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f95fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f95fa7fc640 10 client.?.objecter ms_handle_connect 0x7f96040a96b0 2024-03-25T19:20:21.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.039+0000 7f95fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169183368 0 0) 0x7f9604133a90 con 0x7f9604142a20 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e8004eb0 con 0x7f9604142a20 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206981773 0 0) 0x7f960413b390 con 0x7f96040a96b0 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.041+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9604133a90 con 0x7f96040a96b0 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.042+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635074139 0 0) 0x7f9604139f50 con 0x7f96040b4700 2024-03-25T19:20:21.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.042+0000 7f95fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.042+0000 7f95fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.042+0000 7f95fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.042+0000 7f95fa7fc640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.042+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f960413b390 con 0x7f96040b4700 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.046+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092807552 0 0) 0x7f95e8004eb0 con 0x7f9604142a20 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.046+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9604139f50 con 0x7f9604142a20 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.046+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f95fc001fa0 con 0x7f9604142a20 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.046+0000 7f95fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.046+0000 7f95fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.046+0000 7f95fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.048 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.049 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.049 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.049 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.049 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.049 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.049 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.046+0000 7f95fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.047+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4268627830 0 0) 0x7f9604133a90 con 0x7f96040a96b0 2024-03-25T19:20:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.047+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e8004eb0 con 0x7f96040a96b0 2024-03-25T19:20:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.047+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f95f40018b0 con 0x7f96040a96b0 2024-03-25T19:20:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.047+0000 7f95fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.047+0000 7f95fa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.047+0000 7f95fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:20:21.051 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.048+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3147793173 0 0) 0x7f960413b390 con 0x7f96040b4700 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.048+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9604133a90 con 0x7f96040b4700 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.048+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9600001730 con 0x7f96040b4700 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.048+0000 7f95fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.048+0000 7f95fa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:21.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.048+0000 7f95fa7fc640 10 monclient(hunting): dump: 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2825704646 0 0) 0x7f9604139f50 con 0x7f9604142a20 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 >> v1:172.21.15.60:6790/0 conn(0x7f96040a96b0 legacy=0x7f96041daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 >> v1:172.21.15.90:6789/0 conn(0x7f96040b4700 legacy=0x7f9604132c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:21.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 1 monclient: found mon.a 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960413b520 con 0x7f9604142a20 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f95fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 5 monclient: authenticate success, global_id 4149 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 10 monclient: _renew_subs 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960413c9d0 con 0x7f9604142a20 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 10 monclient: _renew_subs 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96041db330 con 0x7f9604142a20 2024-03-25T19:20:21.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.049+0000 7f960bb15640 1 librados: init done 2024-03-25T19:20:21.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.050+0000 7f960bb15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.050+0000 7f960bb15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.050+0000 7f960bb15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.055 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.050+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960404f730 con 0x7f9604142a20 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95fc003c20 con 0x7f9604142a20 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f95fc003ff0 con 0x7f9604142a20 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 10 monclient: dump: 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.056 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f95fc0045a0 con 0x7f9604142a20 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fbfff640 4 set_mon_vals no callback set 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3071776801 0 0) 0x7f95fc006c90 con 0x7f9604142a20 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 10 client.4149.objecter ms_dispatch 0x7f9604000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:20:21.057 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.053+0000 7f95fa7fc640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T19:20:21.087 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:21.086+0000 7fd81362b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:20:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.154+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f95fc0067c0 con 0x7f9604142a20 2024-03-25T19:20:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.154+0000 7f95fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.155 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.154+0000 7f95fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:21.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.227+0000 7f960bb15640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ae0a333a-7fb1-41ce-b567-a6322ab9b7a8", "id": 5}] 2024-03-25T19:20:21.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.228+0000 7f960bb15640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ae0a333a-7fb1-41ce-b567-a6322ab9b7a8", "id": 5}] 2024-03-25T19:20:21.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.228+0000 7f960bb15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.228+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ae0a333a-7fb1-41ce-b567-a6322ab9b7a8", "id": 5} v 0) v1 -- 0x7f9604069f30 con 0x7f9604142a20 2024-03-25T19:20:21.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.255+0000 7f95fa7fc640 1 -- 172.21.15.90:0/4274450457 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae0a333a-7fb1-41ce-b567-a6322ab9b7a8", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 492759746 0 1988412176) 0x7f95fc006960 con 0x7f9604142a20 2024-03-25T19:20:21.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.256+0000 7f95fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ae0a333a-7fb1-41ce-b567-a6322ab9b7a8", "id": 5}] 2024-03-25T19:20:21.256 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.256+0000 7f95fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:21.256 INFO:teuthology.orchestra.run.smithi090.stdout:5 2024-03-25T19:20:21.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.256+0000 7f960bb15640 10 librados: watch_flush enter 2024-03-25T19:20:21.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.256+0000 7f960bb15640 10 librados: watch_flush exit 2024-03-25T19:20:21.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.256+0000 7f960bb15640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T19:20:21.257 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.256+0000 7f960bb15640 10 client.4149.objecter successfully canceled tick 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 10 monclient: shutdown 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 >> v1:172.21.15.60:6789/0 conn(0x7f9604142a20 legacy=0x7f9604133380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 shutdown_connections 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 >> 172.21.15.90:0/4274450457 conn(0x7f96040668e0 msgr2=0x7f96040b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 shutdown_connections 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 1 -- 172.21.15.90:0/4274450457 wait complete. 2024-03-25T19:20:21.258 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.257+0000 7f960bb15640 1 librados: shutdown 2024-03-25T19:20:21.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 4da3c427-3ae3-460c-afbb-b59f3303db7b 6 2024-03-25T19:20:21.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.368+0000 7fad5dd32640 10 monclient: get_monmap_and_config 2024-03-25T19:20:21.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.368+0000 7fad5dd32640 10 monclient: build_initial_monmap 2024-03-25T19:20:21.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.368+0000 7fad5dd32640 10 monclient: monmap: 2024-03-25T19:20:21.368 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:21.368 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.368 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:21.369824+0000 2024-03-25T19:20:21.369 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:21.369824+0000 2024-03-25T19:20:21.369 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:21.369 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.369 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:21.369 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:21.369 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:21.369 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 1 Processor -- start 2024-03-25T19:20:21.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 1 -- start start 2024-03-25T19:20:21.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient: init 2024-03-25T19:20:21.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:21.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:21.370 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient(hunting): picked mon.noname-c con 0x7fad580b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient(hunting): picked mon.noname-b con 0x7fad58142a20 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient(hunting): picked mon.noname-a con 0x7fad580a96b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580a9d60 con 0x7fad580a96b0 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5813ff10 con 0x7fad580b4700 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580aa1f0 con 0x7fad58142a20 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.369+0000 7fad5dd32640 10 monclient(hunting): authenticate will time out at 6329.643751s 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.370+0000 7fad57fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fad580a96b0 0x7fad5813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39724/0 (socket says 172.21.15.90:39724) 2024-03-25T19:20:21.371 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.370+0000 7fad57fff640 1 -- 172.21.15.90:0/1606562738 learned_addr learned my addr 172.21.15.90:0/1606562738 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:21.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.372+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562636400 0 0) 0x7fad580a9d60 con 0x7fad580a96b0 2024-03-25T19:20:21.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.372+0000 7fad567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.372+0000 7fad567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.372+0000 7fad567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.372+0000 7fad567fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:20:21.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.372+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad40003520 con 0x7fad580a96b0 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843353071 0 0) 0x7fad580aa1f0 con 0x7fad58142a20 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad580a9d60 con 0x7fad58142a20 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3933806669 0 0) 0x7fad5813ff10 con 0x7fad580b4700 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.374 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:20:21.375 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.373+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad580aa1f0 con 0x7fad580b4700 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4156009473 0 0) 0x7fad40003520 con 0x7fad580a96b0 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 >> v1:172.21.15.60:6790/0 conn(0x7fad58142a20 legacy=0x7fad5813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 >> v1:172.21.15.90:6789/0 conn(0x7fad580b4700 legacy=0x7fad58142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 1 monclient: found mon.noname-a 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.376 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad580b4ae0 con 0x7fad580a96b0 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fad50001a10 con 0x7fad580a96b0 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad5dd32640 5 monclient: authenticate success, global_id 4152 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad5dd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad567fc640 10 monclient: dump: 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.377 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.378 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.378 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.378 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.376+0000 7fad5dd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:21.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad50001b70 con 0x7fad580a96b0 2024-03-25T19:20:21.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:21.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad5dd32640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:21.379 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad567fc640 1 -- 172.21.15.90:0/1606562738 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fad50001fc0 con 0x7fad580a96b0 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad5dd32640 4 set_mon_vals no callback set 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.377+0000 7fad567fc640 10 monclient: dump: 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.380 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.378+0000 7fad5dd32640 10 monclient: shutdown 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.378+0000 7fad5dd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.378+0000 7fad5dd32640 1 -- 172.21.15.90:0/1606562738 >> v1:172.21.15.60:6789/0 conn(0x7fad580a96b0 legacy=0x7fad5813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.378+0000 7fad5dd32640 1 -- 172.21.15.90:0/1606562738 shutdown_connections 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.378+0000 7fad5dd32640 1 -- 172.21.15.90:0/1606562738 >> 172.21.15.90:0/1606562738 conn(0x7fad580668e0 msgr2=0x7fad58066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.378+0000 7fad5dd32640 1 -- 172.21.15.90:0/1606562738 shutdown_connections 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.378+0000 7fad5dd32640 1 -- 172.21.15.90:0/1606562738 wait complete. 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 10 monclient: build_initial_monmap 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 10 monclient: monmap: 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:21.381 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 1 librados: starting msgr at 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 1 librados: starting objecter 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 1 Processor -- start 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 1 -- start start 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 1 librados: setting wanted keys 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 1 librados: calling monclient init 2024-03-25T19:20:21.382 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.379+0000 7fad5dd32640 10 monclient: init 2024-03-25T19:20:21.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:21.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:21.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 10 monclient(hunting): picked mon.noname-c con 0x7fad580a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:21.384 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 10 monclient(hunting): picked mon.noname-b con 0x7fad580b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 10 monclient(hunting): picked mon.noname-a con 0x7fad58142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad580abd70 con 0x7fad58142a20 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5813c100 con 0x7fad580b4700 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5813ae50 con 0x7fad580a96b0 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad5dd32640 10 monclient(hunting): authenticate will time out at 6329.654140s 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad57fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fad58142a20 0x7fad580ac190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.90:39726/0 (socket says 172.21.15.90:39726) 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.380+0000 7fad57fff640 1 -- 172.21.15.90:0/2064518223 learned_addr learned my addr 172.21.15.90:0/2064518223 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.381+0000 7fad3ffff640 10 client.?.objecter ms_handle_connect 0x7fad580b4700 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.381+0000 7fad3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.381+0000 7fad3ffff640 10 client.?.objecter ms_handle_connect 0x7fad58142a20 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.381+0000 7fad3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.381+0000 7fad3ffff640 10 client.?.objecter ms_handle_connect 0x7fad580a96b0 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.381+0000 7fad3ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408445580 0 0) 0x7fad580abd70 con 0x7fad58142a20 2024-03-25T19:20:21.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad44004eb0 con 0x7fad58142a20 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928285693 0 0) 0x7fad5813ae50 con 0x7fad580a96b0 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad580abd70 con 0x7fad580a96b0 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527230130 0 0) 0x7fad5813c100 con 0x7fad580b4700 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:20:21.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.383+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5813ae50 con 0x7fad580b4700 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.387+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2220975495 0 0) 0x7fad44004eb0 con 0x7fad58142a20 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.387+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad5813c100 con 0x7fad58142a20 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.387+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fad50001e80 con 0x7fad58142a20 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.387+0000 7fad3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.387+0000 7fad3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.387+0000 7fad3ffff640 10 monclient(hunting): dump: 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.388 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.389 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.389 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.389 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.389 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.387+0000 7fad3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.388+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1638893416 0 0) 0x7fad580abd70 con 0x7fad580a96b0 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.388+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad44004eb0 con 0x7fad580a96b0 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.388+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fad4c001870 con 0x7fad580a96b0 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.388+0000 7fad3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.388+0000 7fad3ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.388+0000 7fad3ffff640 10 monclient(hunting): dump: 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.390 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.390+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 327021203 0 0) 0x7fad5813ae50 con 0x7fad580b4700 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.390+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad580abd70 con 0x7fad580b4700 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.390+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fad480018d0 con 0x7fad580b4700 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.390+0000 7fad3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.390+0000 7fad3ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.390+0000 7fad3ffff640 10 monclient(hunting): dump: 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.391 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.392 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.392 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2690873140 0 0) 0x7fad5813c100 con 0x7fad58142a20 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 >> v1:172.21.15.60:6790/0 conn(0x7fad580a96b0 legacy=0x7fad5813b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 >> v1:172.21.15.90:6789/0 conn(0x7fad580b4700 legacy=0x7fad58139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 1 monclient: found mon.a 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5813cc10 con 0x7fad58142a20 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 10 monclient: _finish_auth 0 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:21.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 5 monclient: authenticate success, global_id 4155 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 10 monclient: _renew_subs 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad581e32a0 con 0x7fad58142a20 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 10 monclient: _renew_subs 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad581e3610 con 0x7fad58142a20 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.391+0000 7fad5dd32640 1 librados: init done 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.392+0000 7fad5dd32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.392+0000 7fad5dd32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.392+0000 7fad5dd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.392+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad580abb70 con 0x7fad58142a20 2024-03-25T19:20:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad50002040 con 0x7fad58142a20 2024-03-25T19:20:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fad50005110 con 0x7fad58142a20 2024-03-25T19:20:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 10 monclient: dump: 2024-03-25T19:20:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fad50005600 con 0x7fad58142a20 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad55ffb640 4 set_mon_vals no callback set 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2166474287 0 0) 0x7fad50006d70 con 0x7fad58142a20 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 10 client.4155.objecter ms_dispatch 0x7fad58000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:20:21.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.395+0000 7fad3ffff640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T19:20:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.494+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fad50006830 con 0x7fad58142a20 2024-03-25T19:20:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.494+0000 7fad3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.494 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.494+0000 7fad3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:21.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.566+0000 7fad5dd32640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4da3c427-3ae3-460c-afbb-b59f3303db7b", "id": 6}] 2024-03-25T19:20:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.566+0000 7fad5dd32640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4da3c427-3ae3-460c-afbb-b59f3303db7b", "id": 6}] 2024-03-25T19:20:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.566+0000 7fad5dd32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:21.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.566+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4da3c427-3ae3-460c-afbb-b59f3303db7b", "id": 6} v 0) v1 -- 0x7fad580b3670 con 0x7fad58142a20 2024-03-25T19:20:21.569 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:21.568+0000 7faf1d632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:20:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.598+0000 7fad3ffff640 1 -- 172.21.15.90:0/2064518223 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4da3c427-3ae3-460c-afbb-b59f3303db7b", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 240021517 0 1113811849) 0x7fad50006a80 con 0x7fad58142a20 2024-03-25T19:20:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.598+0000 7fad3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4da3c427-3ae3-460c-afbb-b59f3303db7b", "id": 6}] 2024-03-25T19:20:21.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.598+0000 7fad3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:21.598 INFO:teuthology.orchestra.run.smithi090.stdout:6 2024-03-25T19:20:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.598+0000 7fad5dd32640 10 librados: watch_flush enter 2024-03-25T19:20:21.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.598+0000 7fad5dd32640 10 librados: watch_flush exit 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.598+0000 7fad5dd32640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.598+0000 7fad5dd32640 10 client.4155.objecter successfully canceled tick 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 10 monclient: shutdown 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 >> v1:172.21.15.60:6789/0 conn(0x7fad58142a20 legacy=0x7fad580ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 shutdown_connections 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 >> 172.21.15.90:0/2064518223 conn(0x7fad580668e0 msgr2=0x7fad5813f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 shutdown_connections 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 1 -- 172.21.15.90:0/2064518223 wait complete. 2024-03-25T19:20:21.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.599+0000 7fad5dd32640 1 librados: shutdown 2024-03-25T19:20:21.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd new 4f96d257-76d5-456f-9a43-180cc136ed55 7 2024-03-25T19:20:21.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.708+0000 7fa5ede4e640 10 monclient: get_monmap_and_config 2024-03-25T19:20:21.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.708+0000 7fa5ede4e640 10 monclient: build_initial_monmap 2024-03-25T19:20:21.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.708+0000 7fa5ede4e640 10 monclient: monmap: 2024-03-25T19:20:21.708 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:21.708 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.708 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:20:21.709553+0000 2024-03-25T19:20:21.708 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:20:21.709553+0000 2024-03-25T19:20:21.709 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:21.709 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.709 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:21.709 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:21.709 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:21.709 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.708+0000 7fa5ede4e640 1 Processor -- start 2024-03-25T19:20:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 1 -- start start 2024-03-25T19:20:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient: init 2024-03-25T19:20:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient(hunting): picked mon.noname-a con 0x7fa5e80b4700 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient(hunting): picked mon.noname-c con 0x7fa5e8142a20 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient(hunting): picked mon.noname-b con 0x7fa5e80a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80a9d60 con 0x7fa5e80b4700 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e813ff10 con 0x7fa5e8142a20 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e80aa1f0 con 0x7fa5e80a96b0 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.709+0000 7fa5ede4e640 10 monclient(hunting): authenticate will time out at 6329.983579s 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.710+0000 7fa5e6ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa5e8142a20 0x7fa5e813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48238/0 (socket says 172.21.15.90:48238) 2024-03-25T19:20:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.710+0000 7fa5e6ffd640 1 -- 172.21.15.90:0/3099886194 learned_addr learned my addr 172.21.15.90:0/3099886194 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.712+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682615756 0 0) 0x7fa5e813ff10 con 0x7fa5e8142a20 2024-03-25T19:20:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.712+0000 7fa5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.712+0000 7fa5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.712+0000 7fa5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.712+0000 7fa5e67fc640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.712+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d0003520 con 0x7fa5e8142a20 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317688555 0 0) 0x7fa5e80aa1f0 con 0x7fa5e80a96b0 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e813ff10 con 0x7fa5e80a96b0 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701883772 0 0) 0x7fa5e80a9d60 con 0x7fa5e80b4700 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:20:21.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.713+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e80aa1f0 con 0x7fa5e80b4700 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225663399 0 0) 0x7fa5d0003520 con 0x7fa5e8142a20 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 >> v1:172.21.15.60:6790/0 conn(0x7fa5e80a96b0 legacy=0x7fa5e813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 >> v1:172.21.15.60:6789/0 conn(0x7fa5e80b4700 legacy=0x7fa5e8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 1 monclient: found mon.noname-c 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80b4ae0 con 0x7fa5e8142a20 2024-03-25T19:20:21.716 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa5dc001690 con 0x7fa5e8142a20 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5ede4e640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5ede4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5e67fc640 10 monclient: dump: 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.717 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.718 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.718 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.718 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.718 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.715+0000 7fa5ede4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5dc0017f0 con 0x7fa5e8142a20 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5e67fc640 1 -- 172.21.15.90:0/3099886194 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa5dc001c00 con 0x7fa5e8142a20 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5e67fc640 10 monclient: dump: 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.719 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5ede4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.717+0000 7fa5ede4e640 4 set_mon_vals no callback set 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.718+0000 7fa5ede4e640 10 monclient: shutdown 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.718+0000 7fa5ede4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.718+0000 7fa5ede4e640 1 -- 172.21.15.90:0/3099886194 >> v1:172.21.15.90:6789/0 conn(0x7fa5e8142a20 legacy=0x7fa5e813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.718+0000 7fa5ede4e640 1 -- 172.21.15.90:0/3099886194 shutdown_connections 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.718+0000 7fa5ede4e640 1 -- 172.21.15.90:0/3099886194 >> 172.21.15.90:0/3099886194 conn(0x7fa5e80668e0 msgr2=0x7fa5e8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:21.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.718+0000 7fa5ede4e640 1 -- 172.21.15.90:0/3099886194 shutdown_connections 2024-03-25T19:20:21.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.718+0000 7fa5ede4e640 1 -- 172.21.15.90:0/3099886194 wait complete. 2024-03-25T19:20:21.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.719+0000 7fa5ede4e640 10 monclient: build_initial_monmap 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.719+0000 7fa5ede4e640 10 monclient: monmap: 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 0 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 0.000000 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:created 0.000000 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 1 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.719+0000 7fa5ede4e640 1 librados: starting msgr at 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.719+0000 7fa5ede4e640 1 librados: starting objecter 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.720+0000 7fa5ede4e640 1 Processor -- start 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.720+0000 7fa5ede4e640 1 -- start start 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.720+0000 7fa5ede4e640 1 librados: setting wanted keys 2024-03-25T19:20:21.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.720+0000 7fa5ede4e640 1 librados: calling monclient init 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.720+0000 7fa5ede4e640 10 monclient: init 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 10 monclient(hunting): picked mon.noname-c con 0x7fa5e80a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 10 monclient(hunting): picked mon.noname-b con 0x7fa5e80b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 10 monclient(hunting): picked mon.noname-a con 0x7fa5e8142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e814eca0 con 0x7fa5e8142a20 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e81d74d0 con 0x7fa5e80b4700 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e81d8350 con 0x7fa5e80a96b0 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5ede4e640 10 monclient(hunting): authenticate will time out at 6329.995063s 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5e6ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa5e80b4700 0x7fa5e80acb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.90:48244/0 (socket says 172.21.15.90:48244) 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5e6ffd640 1 -- 172.21.15.90:0/779287443 learned_addr learned my addr 172.21.15.90:0/779287443 (peer_addr_for_me v1:172.21.15.90:0/0) 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5cffff640 10 client.?.objecter ms_handle_connect 0x7fa5e80b4700 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.721+0000 7fa5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.722+0000 7fa5cffff640 10 client.?.objecter ms_handle_connect 0x7fa5e80a96b0 2024-03-25T19:20:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.722+0000 7fa5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.722+0000 7fa5cffff640 10 client.?.objecter ms_handle_connect 0x7fa5e8142a20 2024-03-25T19:20:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.722+0000 7fa5cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478718776 0 0) 0x7fa5e81d8350 con 0x7fa5e80a96b0 2024-03-25T19:20:21.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): my global_id is 4160 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5d4004eb0 con 0x7fa5e80a96b0 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326053012 0 0) 0x7fa5e81d74d0 con 0x7fa5e80b4700 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e81d8350 con 0x7fa5e80b4700 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683573657 0 0) 0x7fa5e813ff10 con 0x7fa5e8142a20 2024-03-25T19:20:21.726 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:21.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:21.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:21.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.724+0000 7fa5cffff640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:20:21.727 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.725+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e81d74d0 con 0x7fa5e8142a20 2024-03-25T19:20:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3396584909 0 0) 0x7fa5d4004eb0 con 0x7fa5e80a96b0 2024-03-25T19:20:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5e813ff10 con 0x7fa5e80a96b0 2024-03-25T19:20:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2163745801 0 0) 0x7fa5e81d8350 con 0x7fa5e80b4700 2024-03-25T19:20:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5d4004eb0 con 0x7fa5e80b4700 2024-03-25T19:20:21.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa5dc001b40 con 0x7fa5e80b4700 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 10 monclient(hunting): dump: 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.728+0000 7fa5cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.729+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa5d80019b0 con 0x7fa5e80a96b0 2024-03-25T19:20:21.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.729+0000 7fa5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.729+0000 7fa5cffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.729+0000 7fa5cffff640 10 monclient(hunting): dump: 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.731 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.731+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 709874533 0 0) 0x7fa5e81d74d0 con 0x7fa5e8142a20 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.731+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5e81d8350 con 0x7fa5e8142a20 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.731+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa5e00019b0 con 0x7fa5e8142a20 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.731+0000 7fa5cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.731+0000 7fa5cffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.731+0000 7fa5cffff640 10 monclient(hunting): dump: 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.733 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2499185088 0 0) 0x7fa5d4004eb0 con 0x7fa5e80b4700 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 >> v1:172.21.15.60:6790/0 conn(0x7fa5e80a96b0 legacy=0x7fa5e81d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 >> v1:172.21.15.60:6789/0 conn(0x7fa5e8142a20 legacy=0x7fa5e81d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 1 monclient: found mon.b 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80abd90 con 0x7fa5e80b4700 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 10 monclient: _finish_auth 0 2024-03-25T19:20:21.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 10 monclient: _renew_subs 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e81d7720 con 0x7fa5e80b4700 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 10 monclient: _renew_subs 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e80ad240 con 0x7fa5e80b4700 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.732+0000 7fa5ede4e640 1 librados: init done 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.733+0000 7fa5ede4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.733+0000 7fa5ede4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.733+0000 7fa5ede4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:21.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.733+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e804f730 con 0x7fa5e80b4700 2024-03-25T19:20:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5dc0011f0 con 0x7fa5e80b4700 2024-03-25T19:20:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa5dc004c30 con 0x7fa5e80b4700 2024-03-25T19:20:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:21.736 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 10 monclient: dump: 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:epoch 1 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:election_strategy: 3 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa5dc005120 con 0x7fa5e80b4700 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5e5ffb640 4 set_mon_vals no callback set 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2855679264 0 0) 0x7fa5dc006920 con 0x7fa5e80b4700 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 10 client.4156.objecter ms_dispatch 0x7fa5e8000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:20:21.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:20:21.738 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.736+0000 7fa5cffff640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:20:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.827+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa5dc006b30 con 0x7fa5e80b4700 2024-03-25T19:20:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.827+0000 7fa5cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:21.827 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.827+0000 7fa5cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:21.860 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:21.860+0000 7faf1d632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:20:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.901+0000 7fa5ede4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4f96d257-76d5-456f-9a43-180cc136ed55", "id": 7}] 2024-03-25T19:20:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.901+0000 7fa5ede4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4f96d257-76d5-456f-9a43-180cc136ed55", "id": 7}] 2024-03-25T19:20:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.901+0000 7fa5ede4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.901+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4f96d257-76d5-456f-9a43-180cc136ed55", "id": 7} v 0) v1 -- 0x7fa5e80b7470 con 0x7fa5e80b4700 2024-03-25T19:20:21.912 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:21.912+0000 7faf1d632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:20:21.964 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.964+0000 7fa5cffff640 1 -- 172.21.15.90:0/779287443 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f96d257-76d5-456f-9a43-180cc136ed55", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3437343461 0 1371665406) 0x7fa5dc006370 con 0x7fa5e80b4700 2024-03-25T19:20:21.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.964+0000 7fa5cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4f96d257-76d5-456f-9a43-180cc136ed55", "id": 7}] 2024-03-25T19:20:21.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.964+0000 7fa5cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:21.965 INFO:teuthology.orchestra.run.smithi090.stdout:7 2024-03-25T19:20:21.966 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 10 librados: watch_flush enter 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 10 librados: watch_flush exit 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 10 monclient: shutdown 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 >> v1:172.21.15.90:6789/0 conn(0x7fa5e80b4700 legacy=0x7fa5e80acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 shutdown_connections 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 >> 172.21.15.90:0/779287443 conn(0x7fa5e80668e0 msgr2=0x7fa5e80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 shutdown_connections 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 1 -- 172.21.15.90:0/779287443 wait complete. 2024-03-25T19:20:21.967 INFO:teuthology.orchestra.run.smithi090.stderr:2024-03-25T19:20:21.965+0000 7fa5ede4e640 1 librados: shutdown 2024-03-25T19:20:21.968 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:21.968+0000 7fd81362b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:20:21.976 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.976 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:20:21.976 DEBUG:teuthology.orchestra.run.smithi060:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:20:21.978 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:20:21.978 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.979 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:20:21.979 DEBUG:teuthology.orchestra.run.smithi060:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:20:21.981 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:20:21.981 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.981 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:20:21.981 DEBUG:teuthology.orchestra.run.smithi060:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:20:21.983 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:20:21.983 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.983 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:20:21.983 DEBUG:teuthology.orchestra.run.smithi060:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:20:21.985 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:20:21.986 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.986 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:20:21.986 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:20:21.988 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:20:21.988 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.988 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:20:21.988 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:20:21.990 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:20:21.990 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.991 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:20:21.991 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:20:21.993 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:20:21.993 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:20:21.993 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:20:21.993 DEBUG:teuthology.orchestra.run.smithi090:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:20:21.995 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:20:21.995 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:22.042 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:22.041+0000 7faf1d632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.136+0000 7f4cb7f90640 10 monclient: get_monmap_and_config 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.136+0000 7f4cb7f90640 10 monclient: build_initial_monmap 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.136+0000 7f4cb7f90640 10 monclient: monmap: 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:20:22.138189+0000 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:20:22.138189+0000 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:22.137 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:22.138 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:22.138 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.137+0000 7f4cb7f90640 1 Processor -- start 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.137+0000 7f4cb7f90640 1 -- start start 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.137+0000 7f4cb7f90640 10 monclient: init 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb00b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb0142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb00a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00a9d60 con 0x7f4cb0142a20 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb013ff10 con 0x7f4cb00b4700 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00aa1f0 con 0x7f4cb00a96b0 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:22.139 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.138+0000 7f4cb7f90640 10 monclient(hunting): authenticate will time out at 6349.237199s 2024-03-25T19:20:22.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.139+0000 7f4cb652d640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f4cb00a96b0 0x7f4cb013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:33044/0 (socket says 172.21.15.60:33044) 2024-03-25T19:20:22.140 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.139+0000 7f4cb652d640 1 -- 172.21.15.60:0/1364982718 learned_addr learned my addr 172.21.15.60:0/1364982718 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:22.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.142+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873221871 0 0) 0x7f4cb00aa1f0 con 0x7f4cb00a96b0 2024-03-25T19:20:22.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.142+0000 7f4cb4d2a640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.143 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.142+0000 7f4cb4d2a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.142+0000 7f4cb4d2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.142+0000 7f4cb4d2a640 10 monclient(hunting): my global_id is 4163 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.142+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c9c003520 con 0x7f4cb00a96b0 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.143+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723570092 0 0) 0x7f4cb013ff10 con 0x7f4cb00b4700 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.143+0000 7f4cb4d2a640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.143+0000 7f4cb4d2a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.143+0000 7f4cb4d2a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.143+0000 7f4cb4d2a640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:20:22.144 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.143+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb00aa1f0 con 0x7f4cb00b4700 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3246599717 0 0) 0x7f4c9c003520 con 0x7f4cb00a96b0 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 >> v1:172.21.15.90:6789/0 conn(0x7f4cb00b4700 legacy=0x7f4cb0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 >> v1:172.21.15.60:6789/0 conn(0x7f4cb0142a20 legacy=0x7f4cb013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 10 monclient: _finish_hunting 0 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 1 monclient: found mon.noname-b 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00b4ae0 con 0x7f4cb00a96b0 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 10 monclient: _finish_auth 0 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 10 monclient: _check_auth_tickets 2024-03-25T19:20:22.150 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.148+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4ca8001940 con 0x7f4cb00a96b0 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.149+0000 7f4cb4d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.149+0000 7f4cb4d2a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.149+0000 7f4cb4d2a640 10 monclient: dump: 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.149+0000 7f4cb7f90640 5 monclient: authenticate success, global_id 4163 2024-03-25T19:20:22.151 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.149+0000 7f4cb7f90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.151+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ca8001b90 con 0x7f4cb00a96b0 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.151+0000 7f4cb4d2a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb7f90640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb7f90640 4 set_mon_vals no callback set 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb4d2a640 1 -- 172.21.15.60:0/1364982718 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4ca8001f20 con 0x7f4cb00a96b0 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb4d2a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb4d2a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb4d2a640 10 monclient: dump: 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:22.153 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb7f90640 10 monclient: shutdown 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb7f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:22.154 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.152+0000 7f4cb7f90640 1 -- 172.21.15.60:0/1364982718 >> v1:172.21.15.60:6790/0 conn(0x7f4cb00a96b0 legacy=0x7f4cb013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.153+0000 7f4cb7f90640 1 -- 172.21.15.60:0/1364982718 shutdown_connections 2024-03-25T19:20:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.153+0000 7f4cb7f90640 1 -- 172.21.15.60:0/1364982718 >> 172.21.15.60:0/1364982718 conn(0x7f4cb00668e0 msgr2=0x7f4cb0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.154+0000 7f4cb7f90640 1 -- 172.21.15.60:0/1364982718 shutdown_connections 2024-03-25T19:20:22.155 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.154+0000 7f4cb7f90640 1 -- 172.21.15.60:0/1364982718 wait complete. 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.155+0000 7f4cb7f90640 10 monclient: build_initial_monmap 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.155+0000 7f4cb7f90640 10 monclient: monmap: 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 0 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 0.000000 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:created 0.000000 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 1 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:22.156 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:22.157 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.155+0000 7f4cb7f90640 1 librados: starting msgr at 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.155+0000 7f4cb7f90640 1 librados: starting objecter 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.156+0000 7f4cb7f90640 1 Processor -- start 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.156+0000 7f4cb7f90640 1 -- start start 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.156+0000 7f4cb7f90640 1 librados: setting wanted keys 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.156+0000 7f4cb7f90640 1 librados: calling monclient init 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.156+0000 7f4cb7f90640 10 monclient: init 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.157+0000 7f4cb7f90640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.157+0000 7f4cb7f90640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.157+0000 7f4cb7f90640 10 monclient(hunting): picked mon.noname-c con 0x7f4cb00a96b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:22.158 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.157+0000 7f4cb7f90640 10 monclient(hunting): picked mon.noname-b con 0x7f4cb00b4700 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:22.159 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb7f90640 10 monclient(hunting): picked mon.noname-a con 0x7f4cb0142a20 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb7f90640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00ae6d0 con 0x7f4cb0142a20 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb7f90640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00afa00 con 0x7f4cb00b4700 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb7f90640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb00afce0 con 0x7f4cb00a96b0 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb7f90640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb7f90640 10 monclient(hunting): authenticate will time out at 6349.257393s 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb652d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4cb0142a20 0x7f4cb01eee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48050/0 (socket says 172.21.15.60:48050) 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.158+0000 7f4cb652d640 1 -- 172.21.15.60:0/2176747229 learned_addr learned my addr 172.21.15.60:0/2176747229 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.159+0000 7f4ca67fc640 10 client.?.objecter ms_handle_connect 0x7f4cb0142a20 2024-03-25T19:20:22.160 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.159+0000 7f4ca67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.162+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037768455 0 0) 0x7f4cb00ae6d0 con 0x7f4cb0142a20 2024-03-25T19:20:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.162+0000 7f4ca67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.162+0000 7f4ca67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:22.163 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.162+0000 7f4ca67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.162+0000 7f4ca67fc640 10 monclient(hunting): my global_id is 4164 2024-03-25T19:20:22.164 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.162+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c94004eb0 con 0x7f4cb0142a20 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.170+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 961495507 0 0) 0x7f4c94004eb0 con 0x7f4cb0142a20 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.170+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cb00ae6d0 con 0x7f4cb0142a20 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.170+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4ca8001fc0 con 0x7f4cb0142a20 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.170+0000 7f4ca67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.170+0000 7f4ca67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.170+0000 7f4ca67fc640 10 monclient(hunting): dump: 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:22.171 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.170+0000 7f4ca67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.171+0000 7f4ca67fc640 10 client.?.objecter ms_handle_connect 0x7f4cb00b4700 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.171+0000 7f4ca67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:22.172 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.172+0000 7f4ca67fc640 10 client.?.objecter ms_handle_connect 0x7f4cb00a96b0 2024-03-25T19:20:22.173 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.172+0000 7f4ca67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:22.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.184+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1214196839 0 0) 0x7f4cb00ae6d0 con 0x7f4cb0142a20 2024-03-25T19:20:22.184 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.184+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 >> v1:172.21.15.60:6790/0 conn(0x7f4cb00a96b0 legacy=0x7f4cb00b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.186 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 >> v1:172.21.15.90:6789/0 conn(0x7f4cb00b4700 legacy=0x7f4cb00b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 1 monclient: found mon.a 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00ad610 con 0x7f4cb0142a20 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4ca67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 5 monclient: authenticate success, global_id 4164 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 10 monclient: _renew_subs 2024-03-25T19:20:22.187 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb00ae120 con 0x7f4cb0142a20 2024-03-25T19:20:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 10 monclient: _renew_subs 2024-03-25T19:20:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb00b1570 con 0x7f4cb0142a20 2024-03-25T19:20:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.185+0000 7f4cb7f90640 1 librados: init done 2024-03-25T19:20:22.188 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.187+0000 7f4cb7f90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:22.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.187+0000 7f4cb7f90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:22.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.187+0000 7f4cb7f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:22.189 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.187+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb00ae470 con 0x7f4cb0142a20 2024-03-25T19:20:22.417 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ca80014c0 con 0x7f4cb0142a20 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4ca8005030 con 0x7f4cb0142a20 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca67fc640 10 monclient: dump: 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:epoch 1 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:22.418 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:20:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca7fff640 4 set_mon_vals no callback set 2024-03-25T19:20:22.419 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.416+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 746572881 0 0) 0x7f4ca80101d0 con 0x7f4cb0142a20 2024-03-25T19:20:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f4ca8015cd0 con 0x7f4cb0142a20 2024-03-25T19:20:22.421 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 10 client.4164.objecter ms_dispatch 0x7f4cb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 3 client.4164.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 3 client.4164.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:22.422 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 20 client.4164.objecter dump_active .. 0 homeless 2024-03-25T19:20:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4ca8014f90 con 0x7f4cb0142a20 2024-03-25T19:20:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.421+0000 7f4ca67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:22.465 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:22.464+0000 7fd81362b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:20:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.504+0000 7f4cb7f90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:22.504 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.504+0000 7f4cb7f90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:22.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.504+0000 7f4cb7f90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:22.505 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.504+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cb004f730 con 0x7f4cb0142a20 2024-03-25T19:20:22.557 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-03-25T19:20:22.558 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.556+0000 7f4ca67fc640 1 -- 172.21.15.60:0/2176747229 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f4ca8013770 con 0x7f4cb0142a20 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.556+0000 7f4ca67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.556+0000 7f4ca67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.557+0000 7f4cb7f90640 10 librados: watch_flush enter 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 10 librados: watch_flush exit 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 20 client.4164.objecter shutdown clearing up homeless session... 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 10 client.4164.objecter successfully canceled tick 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 10 monclient: shutdown 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 >> v1:172.21.15.60:6789/0 conn(0x7f4cb0142a20 legacy=0x7f4cb01eee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 shutdown_connections 2024-03-25T19:20:22.559 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 >> 172.21.15.60:0/2176747229 conn(0x7f4cb00668e0 msgr2=0x7f4cb00b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 shutdown_connections 2024-03-25T19:20:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 1 -- 172.21.15.60:0/2176747229 wait complete. 2024-03-25T19:20:22.560 INFO:teuthology.orchestra.run.smithi060.stderr:2024-03-25T19:20:22.558+0000 7f4cb7f90640 1 librados: shutdown 2024-03-25T19:20:22.560 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:22.560+0000 7fd81362b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:20:22.573 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:20:22.573 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:20:22.720 INFO:tasks.ceph.mgr.y.smithi060.stderr:2024-03-25T19:20:22.719+0000 7fd81362b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:20:23.379 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.525+0000 7f8ef721e640 10 monclient: get_monmap_and_config 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.525+0000 7f8ef721e640 10 monclient: build_initial_monmap 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.525+0000 7f8ef721e640 10 monclient: monmap: 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:20:23.527471+0000 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:20:23.527471+0000 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:23.526 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:23.527 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:23.527 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:23.527 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:23.527 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:23.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.526+0000 7f8ef721e640 1 Processor -- start 2024-03-25T19:20:23.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.526+0000 7f8ef721e640 1 -- start start 2024-03-25T19:20:23.527 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.526+0000 7f8ef721e640 10 monclient: init 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.527+0000 7f8ef721e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.527+0000 7f8ef721e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.527+0000 7f8ef721e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef00b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.527+0000 7f8ef721e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef00b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef721e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef721e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef00af940 con 0x7f8ef00b1730 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef721e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef00b0a70 con 0x7f8ef00b4d60 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef721e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef00b0d50 con 0x7f8ef014be90 2024-03-25T19:20:23.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef721e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:23.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef721e640 10 monclient(hunting): authenticate will time out at 6350.626982s 2024-03-25T19:20:23.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef57bb640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f8ef014be90 0x7f8ef014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:33056/0 (socket says 172.21.15.60:33056) 2024-03-25T19:20:23.530 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.528+0000 7f8ef57bb640 1 -- 172.21.15.60:0/797894359 learned_addr learned my addr 172.21.15.60:0/797894359 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:23.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.806+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561287091 0 0) 0x7f8ef00af940 con 0x7f8ef00b1730 2024-03-25T19:20:23.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.806+0000 7f8eef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:20:23.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.806+0000 7f8eef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:23.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.806+0000 7f8eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:23.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.806+0000 7f8eef7fe640 10 monclient(hunting): my global_id is 4179 2024-03-25T19:20:23.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.806+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee0003520 con 0x7f8ef00b1730 2024-03-25T19:20:23.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.811+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1821832199 0 0) 0x7f8ee0003520 con 0x7f8ef00b1730 2024-03-25T19:20:23.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.811+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 >> v1:172.21.15.60:6790/0 conn(0x7f8ef014be90 legacy=0x7f8ef014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:23.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.811+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 >> v1:172.21.15.90:6789/0 conn(0x7f8ef00b4d60 legacy=0x7f8ef013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:23.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.811+0000 7f8eef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:20:23.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.811+0000 7f8eef7fe640 1 monclient: found mon.noname-a 2024-03-25T19:20:23.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00b1030 con 0x7f8ef00b1730 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8edc001690 con 0x7f8ef00b1730 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8eef7fe640 10 monclient: dump: 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:23.813 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:23.814 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:23.814 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:23.814 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:23.814 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8ef721e640 5 monclient: authenticate success, global_id 4179 2024-03-25T19:20:23.814 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.812+0000 7f8ef721e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:23.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8edc0017f0 con 0x7f8ef00b1730 2024-03-25T19:20:23.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8eef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:23.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8eef7fe640 1 -- 172.21.15.60:0/797894359 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8edc001c00 con 0x7f8ef00b1730 2024-03-25T19:20:23.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:23.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8eef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:23.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8eef7fe640 10 monclient: dump: 2024-03-25T19:20:23.815 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8ef721e640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:23.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.815+0000 7f8ef721e640 4 set_mon_vals no callback set 2024-03-25T19:20:23.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.816+0000 7f8ef721e640 10 monclient: shutdown 2024-03-25T19:20:23.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.816+0000 7f8ef721e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:23.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.816+0000 7f8ef721e640 1 -- 172.21.15.60:0/797894359 >> v1:172.21.15.60:6789/0 conn(0x7f8ef00b1730 legacy=0x7f8ef00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:23.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.816+0000 7f8ef721e640 1 -- 172.21.15.60:0/797894359 shutdown_connections 2024-03-25T19:20:23.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.816+0000 7f8ef721e640 1 -- 172.21.15.60:0/797894359 >> 172.21.15.60:0/797894359 conn(0x7f8ef00617a0 msgr2=0x7f8ef0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:23.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.816+0000 7f8ef721e640 1 -- 172.21.15.60:0/797894359 shutdown_connections 2024-03-25T19:20:23.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.816+0000 7f8ef721e640 1 -- 172.21.15.60:0/797894359 wait complete. 2024-03-25T19:20:23.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.817+0000 7f8ef721e640 10 monclient: build_initial_monmap 2024-03-25T19:20:23.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.817+0000 7f8ef721e640 10 monclient: monmap: 2024-03-25T19:20:23.818 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:23.818 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:23.818 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:20:23.818 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.817+0000 7f8ef721e640 1 librados: starting msgr at 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.817+0000 7f8ef721e640 1 librados: starting objecter 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.817+0000 7f8ef721e640 1 Processor -- start 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.818+0000 7f8ef721e640 1 -- start start 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.818+0000 7f8ef721e640 1 librados: setting wanted keys 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.818+0000 7f8ef721e640 1 librados: calling monclient init 2024-03-25T19:20:23.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.818+0000 7f8ef721e640 10 monclient: init 2024-03-25T19:20:23.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ef00b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ef00b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ef014be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef01d7940 con 0x7f8ef00b4d60 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef01d8570 con 0x7f8ef014be90 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef01d8700 con 0x7f8ef00b1730 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:23.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.819+0000 7f8ef721e640 10 monclient(hunting): authenticate will time out at 6350.918625s 2024-03-25T19:20:23.828 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.828+0000 7f8eeffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8ef00b4d60 0x7f8ef0136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:48060/0 (socket says 172.21.15.60:48060) 2024-03-25T19:20:23.828 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.828+0000 7f8eeffff640 1 -- 172.21.15.60:0/136389276 learned_addr learned my addr 172.21.15.60:0/136389276 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:23.876 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.876+0000 7f8eed7fa640 10 client.?.objecter ms_handle_connect 0x7f8ef00b4d60 2024-03-25T19:20:23.876 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.876+0000 7f8eed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:23.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.886+0000 7f8eed7fa640 10 client.?.objecter ms_handle_connect 0x7f8ef014be90 2024-03-25T19:20:23.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.886+0000 7f8eed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:23.898 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.898+0000 7f8eed7fa640 10 client.?.objecter ms_handle_connect 0x7f8ef00b1730 2024-03-25T19:20:23.898 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.898+0000 7f8eed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:23.899 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:23.899+0000 7faebdc10640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:20:23.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.902+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027033314 0 0) 0x7f8ef01d8570 con 0x7f8ef014be90 2024-03-25T19:20:23.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.902+0000 7f8eed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:23.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.902+0000 7f8eed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:23.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.902+0000 7f8eed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:23.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.902+0000 7f8eed7fa640 10 monclient(hunting): my global_id is 4168 2024-03-25T19:20:23.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.902+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed4004eb0 con 0x7f8ef014be90 2024-03-25T19:20:23.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.904+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113028496 0 0) 0x7f8ef01d7940 con 0x7f8ef00b4d60 2024-03-25T19:20:23.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.904+0000 7f8eed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:23.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.904+0000 7f8eed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:23.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.904+0000 7f8eed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:23.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.904+0000 7f8eed7fa640 10 monclient(hunting): my global_id is 4182 2024-03-25T19:20:23.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.904+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef01d8570 con 0x7f8ef00b4d60 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.910+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3848008849 0 0) 0x7f8ed4004eb0 con 0x7f8ef014be90 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.910+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef01d7940 con 0x7f8ef014be90 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.910+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8ee8001b00 con 0x7f8ef014be90 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.910+0000 7f8eed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.910+0000 7f8eed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.910+0000 7f8eed7fa640 10 monclient(hunting): dump: 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.911 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.912 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:23.912 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:23.912 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:23.912 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:23.912 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:23.912 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:23.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.910+0000 7f8eed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.915+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023416809 0 0) 0x7f8ef01d8700 con 0x7f8ef00b1730 2024-03-25T19:20:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.915+0000 7f8eed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:20:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.915+0000 7f8eed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.915+0000 7f8eed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:23.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.915+0000 7f8eed7fa640 10 monclient(hunting): my global_id is 4178 2024-03-25T19:20:23.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.915+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed4004eb0 con 0x7f8ef00b1730 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.938+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1241817402 0 0) 0x7f8ef01d8570 con 0x7f8ef00b4d60 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.938+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef01d8700 con 0x7f8ef00b4d60 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.938+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8edc001bc0 con 0x7f8ef00b4d60 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.938+0000 7f8eed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.938+0000 7f8eed7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.938+0000 7f8eed7fa640 10 monclient(hunting): dump: 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:23.939 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:23.940 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1066392116 0 0) 0x7f8ef01d8700 con 0x7f8ef00b4d60 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 >> v1:172.21.15.60:6790/0 conn(0x7f8ef00b1730 legacy=0x7f8ef01d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 >> v1:172.21.15.90:6789/0 conn(0x7f8ef014be90 legacy=0x7f8ef012fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 1 monclient: found mon.a 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef01361c0 con 0x7f8ef00b4d60 2024-03-25T19:20:23.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 10 monclient: _finish_auth 0 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8eed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 5 monclient: authenticate success, global_id 4182 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 10 monclient: _renew_subs 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef0136470 con 0x7f8ef00b4d60 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 10 client.4182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 10 monclient: _renew_subs 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef0137780 con 0x7f8ef00b4d60 2024-03-25T19:20:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.944+0000 7f8ef721e640 1 librados: init done 2024-03-25T19:20:23.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.946+0000 7f8edaffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:23.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.946+0000 7f8edaffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:23.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.946+0000 7f8edaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:23.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:23.946+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0005080 con 0x7f8ef00b4d60 2024-03-25T19:20:23.971 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:20:23.971+0000 7faebdc10640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.416+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8edc0038a0 con 0x7f8ef00b4d60 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.416+0000 7f8eed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.416+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8edc003c70 con 0x7f8ef00b4d60 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.416+0000 7f8eed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.416+0000 7f8eed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.416+0000 7f8eed7fa640 10 monclient: dump: 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:24.417 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:24.418 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:24.418 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:24.418 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:24.418 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:24.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.416+0000 7f8eeeffd640 4 set_mon_vals no callback set 2024-03-25T19:20:24.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.417+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2036724358 0 0) 0x7f8edc003e90 con 0x7f8ef00b4d60 2024-03-25T19:20:24.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.418+0000 7f8ef4fba640 10 monclient: get_auth_request con 0x7f8ed406de50 auth_method 0 2024-03-25T19:20:24.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.418+0000 7f8eed7fa640 10 client.4182.objecter ms_handle_connect 0x7f8ed406de50 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f8edc054880 con 0x7f8ef00b4d60 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 10 client.4182.objecter ms_dispatch 0x7f8ef0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 3 client.4182.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 3 client.4182.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 20 client.4182.objecter dump_active .. 0 homeless 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8edc0b18b0 con 0x7f8ef00b4d60 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:24.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.425+0000 7f8eed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:24.796 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.795+0000 7f8edaffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:24.796 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.795+0000 7f8edaffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:24.796 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.795+0000 7f8edaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:24.796 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.795+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ed0005330 con 0x7f8ef00b4d60 2024-03-25T19:20:24.834 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.833+0000 7f8eed7fa640 1 -- 172.21.15.60:0/136389276 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f8edc050390 con 0x7f8ef00b4d60 2024-03-25T19:20:24.834 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.834+0000 7f8eed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:24.834 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.834+0000 7f8eed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:24.835 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:20:24.835 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 10 librados: watch_flush enter 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 10 librados: watch_flush exit 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 20 client.4182.objecter shutdown clearing up homeless session... 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 10 client.4182.objecter successfully canceled tick 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 >> v1:172.21.15.90:6800/1827638754 conn(0x7f8ed406de50 legacy=0x7f8ed4070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 10 monclient: shutdown 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 >> v1:172.21.15.60:6789/0 conn(0x7f8ef00b4d60 legacy=0x7f8ef0136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.836+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 shutdown_connections 2024-03-25T19:20:24.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.837+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 >> 172.21.15.60:0/136389276 conn(0x7f8ef00617a0 msgr2=0x7f8ef0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:24.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.837+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 shutdown_connections 2024-03-25T19:20:24.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.837+0000 7f8edaffd640 1 -- 172.21.15.60:0/136389276 wait complete. 2024-03-25T19:20:24.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:24.837+0000 7f8edaffd640 1 librados: shutdown 2024-03-25T19:20:24.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:28.902 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:06.842 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:28.926 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:06.873 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:28.962 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:06.902 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:28.986 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:06.933 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.038 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:06.970 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.093 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:07.018 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.097 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:07.029 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.154 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:07.079 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.453 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:07.381 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.481 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:07.431 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.509 INFO:tasks.ceph.osd.7.smithi090.stderr:2024-03-25T19:20:29.501+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:29.515 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:07.443 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.537 INFO:tasks.ceph.osd.4.smithi090.stderr:2024-03-25T19:20:29.531+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:29.545 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:07.496 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.606 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:07.537 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.665 INFO:tasks.ceph.osd.5.smithi090.stderr:2024-03-25T19:20:29.659+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:29.668 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:07.599 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:29.754 INFO:tasks.ceph.osd.6.smithi090.stderr:2024-03-25T19:20:29.746+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:29.960 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:07.900 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.030 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:07.961 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.043 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:07.990 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.087 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-25T19:20:30.082+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:30.091 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-25T19:20:30.086+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:30.091 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:08.022 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.123 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:08.055 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.234 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:08.158 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.236 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-25T19:20:30.231+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:30.332 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:08.272 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.411 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:08.358 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.490 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:08.421 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.561 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:08.511 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.566 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:08.494 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.595 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:08.521 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.636 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-25T19:20:30.630+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:20:30.654 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:08.594 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.720 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:08.651 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.763 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:08.710 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.808 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:08.740 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.907 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:08.832 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.926 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:08.867 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.937 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:08.888 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:30.947 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:08.875 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.051 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:08.998 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.083 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:09.015 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.102 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:09.033 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.131 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:09.062 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.178 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:09.103 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.196 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:09.136 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.309 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:09.259 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.320 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:09.247 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.339 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:09.286 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.358 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:09.290 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.428 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:09.359 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.442 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:09.367 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.466 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:09.407 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.510 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:09.441 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.598 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:09.548 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.626 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:09.558 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.630 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:09.577 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.631 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:09.559 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.655 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:31.699 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:09.629 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.706 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:09.632 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.730 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:09.671 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.744+0000 7eff27c2a640 10 monclient: get_monmap_and_config 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.744+0000 7eff27c2a640 10 monclient: build_initial_monmap 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.744+0000 7eff27c2a640 10 monclient: monmap: 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:20:31.745214+0000 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:20:31.745214+0000 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:31.744 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 1 Processor -- start 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 1 -- start start 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient: init 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient(hunting): picked mon.noname-a con 0x7eff200b10d0 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient(hunting): picked mon.noname-c con 0x7eff200b4360 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient(hunting): picked mon.noname-b con 0x7eff200a9470 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff200a9ab0 con 0x7eff200b10d0 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff200b2400 con 0x7eff200b4360 2024-03-25T19:20:31.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff200aa2a0 con 0x7eff200a9470 2024-03-25T19:20:31.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:31.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.745+0000 7eff27c2a640 10 monclient(hunting): authenticate will time out at 6358.843861s 2024-03-25T19:20:31.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.746+0000 7eff261c7640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7eff200a9470 0x7eff2014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:45206/0 (socket says 172.21.15.60:45206) 2024-03-25T19:20:31.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.746+0000 7eff261c7640 1 -- 172.21.15.60:0/3170905461 learned_addr learned my addr 172.21.15.60:0/3170905461 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:31.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.748+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701234331 0 0) 0x7eff200aa2a0 con 0x7eff200a9470 2024-03-25T19:20:31.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.748+0000 7eff249c4640 10 monclient(hunting): _negotiate 2024-03-25T19:20:31.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.748+0000 7eff249c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:31.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.748+0000 7eff249c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.748+0000 7eff249c4640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.748+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff0c003520 con 0x7eff200a9470 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384547465 0 0) 0x7eff200a9ab0 con 0x7eff200b10d0 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): _negotiate 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff200aa2a0 con 0x7eff200b10d0 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200952374 0 0) 0x7eff200b2400 con 0x7eff200b4360 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): _negotiate 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 10 monclient(hunting): my global_id is 4195 2024-03-25T19:20:31.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.749+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff200a9ab0 con 0x7eff200b4360 2024-03-25T19:20:31.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.751+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 717415752 0 0) 0x7eff0c003520 con 0x7eff200a9470 2024-03-25T19:20:31.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.751+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 >> v1:172.21.15.90:6789/0 conn(0x7eff200b4360 legacy=0x7eff200b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:31.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.751+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 >> v1:172.21.15.60:6789/0 conn(0x7eff200b10d0 legacy=0x7eff200b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:31.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.751+0000 7eff249c4640 10 monclient: _finish_hunting 0 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.751+0000 7eff249c4640 1 monclient: found mon.noname-b 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.751+0000 7eff249c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff200b14b0 con 0x7eff200a9470 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 10 monclient: _finish_auth 0 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 10 monclient: _check_auth_tickets 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7eff18001a10 con 0x7eff200a9470 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff27c2a640 5 monclient: authenticate success, global_id 4205 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff27c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff249c4640 10 monclient: dump: 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:31.752 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:31.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.752+0000 7eff27c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff18001b70 con 0x7eff200a9470 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff249c4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff249c4640 1 -- 172.21.15.60:0/3170905461 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7eff18001f80 con 0x7eff200a9470 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff27c2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff27c2a640 4 set_mon_vals no callback set 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff249c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff249c4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.753+0000 7eff249c4640 10 monclient: dump: 2024-03-25T19:20:31.756 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:31.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 10 monclient: shutdown 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 1 -- 172.21.15.60:0/3170905461 >> v1:172.21.15.60:6790/0 conn(0x7eff200a9470 legacy=0x7eff2014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 1 -- 172.21.15.60:0/3170905461 shutdown_connections 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 1 -- 172.21.15.60:0/3170905461 >> 172.21.15.60:0/3170905461 conn(0x7eff200617a0 msgr2=0x7eff20061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 1 -- 172.21.15.60:0/3170905461 shutdown_connections 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 1 -- 172.21.15.60:0/3170905461 wait complete. 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 10 monclient: build_initial_monmap 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.754+0000 7eff27c2a640 10 monclient: monmap: 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:20:31.758 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 librados: starting msgr at 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 librados: starting objecter 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 Processor -- start 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 -- start start 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 librados: setting wanted keys 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 librados: calling monclient init 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient: init 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:20:31.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient(hunting): picked mon.noname-b con 0x7eff200a9470 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient(hunting): picked mon.noname-c con 0x7eff200b10d0 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient(hunting): picked mon.noname-a con 0x7eff200b4360 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff201d7490 con 0x7eff200b4360 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff200ac970 con 0x7eff200a9470 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff201d76c0 con 0x7eff200b10d0 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.755+0000 7eff27c2a640 10 monclient(hunting): authenticate will time out at 6358.853815s 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff251c5640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7eff200b10d0 0x7eff20138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:45210/0 (socket says 172.21.15.60:45210) 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff251c5640 1 -- 172.21.15.60:0/3144029438 learned_addr learned my addr 172.21.15.60:0/3144029438 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff167fc640 10 client.?.objecter ms_handle_connect 0x7eff200b10d0 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff167fc640 10 client.?.objecter ms_handle_connect 0x7eff200a9470 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff167fc640 10 client.?.objecter ms_handle_connect 0x7eff200b4360 2024-03-25T19:20:31.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.756+0000 7eff167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:31.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402760142 0 0) 0x7eff200ac970 con 0x7eff200a9470 2024-03-25T19:20:31.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): my global_id is 4198 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff04004eb0 con 0x7eff200a9470 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598171287 0 0) 0x7eff201d76c0 con 0x7eff200b10d0 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff200ac970 con 0x7eff200b10d0 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761236843 0 0) 0x7eff201d7490 con 0x7eff200b4360 2024-03-25T19:20:31.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:20:31.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:31.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:31.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:20:31.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.758+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff201d76c0 con 0x7eff200b4360 2024-03-25T19:20:31.764 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.762+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834583203 0 0) 0x7eff04004eb0 con 0x7eff200a9470 2024-03-25T19:20:31.764 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.762+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff201d7490 con 0x7eff200a9470 2024-03-25T19:20:31.764 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.762+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7eff1c001970 con 0x7eff200a9470 2024-03-25T19:20:31.764 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.762+0000 7eff167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:31.764 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:31.764 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): dump: 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 628954614 0 0) 0x7eff200ac970 con 0x7eff200b10d0 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff04004eb0 con 0x7eff200b10d0 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7eff100019b0 con 0x7eff200b10d0 2024-03-25T19:20:31.765 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): dump: 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 27118708 0 0) 0x7eff201d76c0 con 0x7eff200b4360 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff200ac970 con 0x7eff200b4360 2024-03-25T19:20:31.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7eff18001e80 con 0x7eff200b4360 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.763+0000 7eff167fc640 10 monclient(hunting): dump: 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:31.767 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 130656392 0 0) 0x7eff04004eb0 con 0x7eff200b10d0 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 >> v1:172.21.15.90:6789/0 conn(0x7eff200a9470 legacy=0x7eff201d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 >> v1:172.21.15.60:6789/0 conn(0x7eff200b4360 legacy=0x7eff201e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 1 monclient: found mon.c 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff201d82f0 con 0x7eff200b10d0 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 10 monclient: _finish_auth 0 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 5 monclient: authenticate success, global_id 4208 2024-03-25T19:20:31.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 10 monclient: _renew_subs 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff200ab8a0 con 0x7eff200b10d0 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 10 monclient: _renew_subs 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2013ea40 con 0x7eff200b10d0 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.765+0000 7eff27c2a640 1 librados: init done 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.767+0000 7eff27c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.767+0000 7eff27c2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.767+0000 7eff27c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:31.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.767+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff200627b0 con 0x7eff200b10d0 2024-03-25T19:20:31.824 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:09.755 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.886 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:09.837 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.894 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:09.826 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.924 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:09.852 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.931 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:09.878 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.970 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:09.900 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.971 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:09.896 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:31.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.980+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff10003610 con 0x7eff200b10d0 2024-03-25T19:20:31.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.980+0000 7eff167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:31.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:31.980+0000 7eff17fff640 4 set_mon_vals no callback set 2024-03-25T19:20:31.995 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:09.935 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.089 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:10.019 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.102+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7eff100039a0 con 0x7eff200b10d0 2024-03-25T19:20:32.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.102+0000 7eff167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:32.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.102+0000 7eff167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:32.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.102+0000 7eff167fc640 10 monclient: dump: 2024-03-25T19:20:32.101 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:32.101 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:32.101 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:32.102 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:32.102 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:32.102 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:32.102 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:32.102 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:32.102 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:32.102 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:32.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.102+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7eff10003b80 con 0x7eff200b10d0 2024-03-25T19:20:32.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.103+0000 7eff259c6640 10 monclient: get_auth_request con 0x7eff0406e160 auth_method 0 2024-03-25T19:20:32.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.103+0000 7eff167fc640 10 client.4208.objecter ms_handle_connect 0x7eff0406e160 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7eff100547b0 con 0x7eff200b10d0 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 10 client.4208.objecter ms_dispatch 0x7eff20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff100502c0 con 0x7eff200b10d0 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:32.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.109+0000 7eff167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:32.175 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:10.126 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.176 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:10.108 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.218 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:10.165 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.220 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:10.148 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.239 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:10.164 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.248 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:10.179 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.275 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:10.216 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.293+0000 7eff27c2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:32.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.293+0000 7eff27c2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:32.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.293+0000 7eff27c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:32.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.293+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff201d7850 con 0x7eff200b10d0 2024-03-25T19:20:32.352 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:10.282 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.360+0000 7eff167fc640 1 -- 172.21.15.60:0/3144029438 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7eff100521f0 con 0x7eff200b10d0 2024-03-25T19:20:32.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.360+0000 7eff167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:32.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.360+0000 7eff167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:32.360 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:20:32.360 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 10 librados: watch_flush enter 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 10 librados: watch_flush exit 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 10 client.4208.objecter successfully canceled tick 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 >> v1:172.21.15.90:6800/1827638754 conn(0x7eff0406e160 legacy=0x7eff04070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 10 monclient: shutdown 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.362+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 >> v1:172.21.15.60:6790/0 conn(0x7eff200b10d0 legacy=0x7eff20138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.363+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 shutdown_connections 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.363+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 >> 172.21.15.60:0/3144029438 conn(0x7eff200617a0 msgr2=0x7eff20145e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.363+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 shutdown_connections 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.363+0000 7eff27c2a640 1 -- 172.21.15.60:0/3144029438 wait complete. 2024-03-25T19:20:32.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:32.363+0000 7eff27c2a640 1 librados: shutdown 2024-03-25T19:20:32.372 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:32.455 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:10.387 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.465 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:10.416 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.514 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:10.439 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.521 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:10.468 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.522 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:10.450 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.523 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:10.453 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.616 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:10.547 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.755 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:10.706 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.757 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:10.697 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.799 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:10.729 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.815 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:10.755 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.827 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:10.755 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.889 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:10.819 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.976 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:10.908 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:32.990 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:10.915 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.031 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:10.963 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.044 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:10.995 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.045 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:10.969 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.046 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:10.993 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.065 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:10.997 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.121 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:11.048 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.133 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:11.061 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.170 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:11.100 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.446 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:11.376 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.558 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:11.489 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.581 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:11.531 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.614 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:11.545 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.637 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:11.587 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.680 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:11.608 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.736 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:11.663 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:33.957 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:11.888 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:34.013 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:11.944 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:38.442 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:16.381 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:38.572 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:16.504 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:38.599 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:16.546 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:38.624 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:16.549 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:39.177 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:39.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.300+0000 7f941646d640 10 monclient: get_monmap_and_config 2024-03-25T19:20:39.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.300+0000 7f941646d640 10 monclient: build_initial_monmap 2024-03-25T19:20:39.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.300+0000 7f941646d640 10 monclient: monmap: 2024-03-25T19:20:39.299 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:20:39.301033+0000 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:20:39.301033+0000 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:39.300 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 1 Processor -- start 2024-03-25T19:20:39.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 1 -- start start 2024-03-25T19:20:39.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient: init 2024-03-25T19:20:39.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient(hunting): picked mon.noname-b con 0x7f94100b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient(hunting): picked mon.noname-c con 0x7f94100b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient(hunting): picked mon.noname-a con 0x7f941014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94100af940 con 0x7f941014be90 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94100b0a70 con 0x7f94100b1730 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94100b0d50 con 0x7f94100b4d60 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.301+0000 7f941646d640 10 monclient(hunting): authenticate will time out at 6366.399580s 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.302+0000 7f940ffff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f94100b4d60 0x7f941013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:37036/0 (socket says 172.21.15.60:37036) 2024-03-25T19:20:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.302+0000 7f940ffff640 1 -- 172.21.15.60:0/453991862 learned_addr learned my addr 172.21.15.60:0/453991862 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840028450 0 0) 0x7f94100af940 con 0x7f941014be90 2024-03-25T19:20:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:20:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f8003520 con 0x7f941014be90 2024-03-25T19:20:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817211531 0 0) 0x7f94100b0d50 con 0x7f94100b4d60 2024-03-25T19:20:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94100af940 con 0x7f94100b4d60 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549142961 0 0) 0x7f94100b0a70 con 0x7f94100b1730 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _negotiate 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 10 monclient(hunting): my global_id is 4201 2024-03-25T19:20:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.305+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94100b0d50 con 0x7f94100b1730 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 355083997 0 0) 0x7f93f8003520 con 0x7f941014be90 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 >> v1:172.21.15.60:6790/0 conn(0x7f94100b4d60 legacy=0x7f941013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 >> v1:172.21.15.90:6789/0 conn(0x7f94100b1730 legacy=0x7f94100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 10 monclient: _finish_hunting 0 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 1 monclient: found mon.noname-a 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94100b1030 con 0x7f941014be90 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 10 monclient: _finish_auth 0 2024-03-25T19:20:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 10 monclient: _check_auth_tickets 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9408001a10 con 0x7f941014be90 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f940effd640 10 monclient: dump: 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:39.310 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:39.311 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f941646d640 5 monclient: authenticate success, global_id 4215 2024-03-25T19:20:39.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.309+0000 7f941646d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:39.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9408001b70 con 0x7f941014be90 2024-03-25T19:20:39.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f940effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f940effd640 1 -- 172.21.15.60:0/453991862 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9408001f80 con 0x7f941014be90 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f941646d640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f941646d640 4 set_mon_vals no callback set 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f940effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f940effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f940effd640 10 monclient: dump: 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:39.313 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f941646d640 10 monclient: shutdown 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f941646d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.311+0000 7f941646d640 1 -- 172.21.15.60:0/453991862 >> v1:172.21.15.60:6789/0 conn(0x7f941014be90 legacy=0x7f941014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.312+0000 7f941646d640 1 -- 172.21.15.60:0/453991862 shutdown_connections 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.312+0000 7f941646d640 1 -- 172.21.15.60:0/453991862 >> 172.21.15.60:0/453991862 conn(0x7f94100617a0 msgr2=0x7f9410061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.312+0000 7f941646d640 1 -- 172.21.15.60:0/453991862 shutdown_connections 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.312+0000 7f941646d640 1 -- 172.21.15.60:0/453991862 wait complete. 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.312+0000 7f941646d640 10 monclient: build_initial_monmap 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.312+0000 7f941646d640 10 monclient: monmap: 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:20:39.314 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 librados: starting msgr at 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 librados: starting objecter 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 Processor -- start 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 -- start start 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 librados: setting wanted keys 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 librados: calling monclient init 2024-03-25T19:20:39.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient: init 2024-03-25T19:20:39.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:39.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:39.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient(hunting): picked mon.noname-c con 0x7f94100b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:39.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient(hunting): picked mon.noname-a con 0x7f94100b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:39.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient(hunting): picked mon.noname-b con 0x7f941014be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:39.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94101d87c0 con 0x7f94100b4d60 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9410138820 con 0x7f941014be90 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94101389b0 con 0x7f94100b1730 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.313+0000 7f941646d640 10 monclient(hunting): authenticate will time out at 6366.411839s 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940f7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f94100b4d60 0x7f94101d80b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47748/0 (socket says 172.21.15.60:47748) 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940f7fe640 1 -- 172.21.15.60:0/3445379482 learned_addr learned my addr 172.21.15.60:0/3445379482 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940cff9640 10 client.?.objecter ms_handle_connect 0x7f941014be90 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940cff9640 10 client.?.objecter ms_handle_connect 0x7f94100b4d60 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940cff9640 10 client.?.objecter ms_handle_connect 0x7f94100b1730 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.314+0000 7f940cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344473953 0 0) 0x7f94101389b0 con 0x7f94100b1730 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:39.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93fc004eb0 con 0x7f94100b1730 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102183676 0 0) 0x7f9410138820 con 0x7f941014be90 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): my global_id is 4204 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94101389b0 con 0x7f941014be90 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086711682 0 0) 0x7f94101d87c0 con 0x7f94100b4d60 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:39.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:20:39.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.317+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9410138820 con 0x7f94100b4d60 2024-03-25T19:20:39.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.323+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837998245 0 0) 0x7f94101389b0 con 0x7f941014be90 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.323+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94101d87c0 con 0x7f941014be90 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9408001e80 con 0x7f941014be90 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): dump: 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:39.324 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2877544233 0 0) 0x7f9410138820 con 0x7f94100b4d60 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94101389b0 con 0x7f94100b4d60 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f94040019b0 con 0x7f94100b4d60 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): dump: 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:39.325 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 652504084 0 0) 0x7f93fc004eb0 con 0x7f94100b1730 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9410138820 con 0x7f94100b1730 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f94000018b0 con 0x7f94100b1730 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.324+0000 7f940cff9640 10 monclient(hunting): dump: 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.326 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:39.327 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:39.327 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:39.327 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:39.327 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:39.327 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4048056546 0 0) 0x7f9410138820 con 0x7f94100b1730 2024-03-25T19:20:39.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 >> v1:172.21.15.90:6789/0 conn(0x7f941014be90 legacy=0x7f94101e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:39.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 >> v1:172.21.15.60:6789/0 conn(0x7f94100b4d60 legacy=0x7f94101d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 1 monclient: found mon.c 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94101405f0 con 0x7f94100b1730 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 10 monclient: _finish_auth 0 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f940cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 5 monclient: authenticate success, global_id 4214 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 10 monclient: _renew_subs 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94101407e0 con 0x7f94100b1730 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:39.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 10 monclient: _renew_subs 2024-03-25T19:20:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94101e3ba0 con 0x7f94100b1730 2024-03-25T19:20:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.328+0000 7f941646d640 1 librados: init done 2024-03-25T19:20:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.330+0000 7f941646d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.330+0000 7f941646d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.330+0000 7f941646d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.330+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94101d7f00 con 0x7f94100b1730 2024-03-25T19:20:39.331 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:17.257 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:39.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.331+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9400001d40 con 0x7f94100b1730 2024-03-25T19:20:39.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.331+0000 7f940cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:39.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.331+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9400004a00 con 0x7f94100b1730 2024-03-25T19:20:39.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.331+0000 7f940e7fc640 4 set_mon_vals no callback set 2024-03-25T19:20:39.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.331+0000 7f940cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:39.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.331+0000 7f940cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:39.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.331+0000 7f940cff9640 10 monclient: dump: 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:39.333 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:39.343 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:17.294 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:39.370 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:17.301 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:39.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.588+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f940001a9d0 con 0x7f94100b1730 2024-03-25T19:20:39.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.589+0000 7f940f7fe640 10 monclient: get_auth_request con 0x7f93fc06e0c0 auth_method 0 2024-03-25T19:20:39.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.589+0000 7f940cff9640 10 client.4214.objecter ms_handle_connect 0x7f93fc06e0c0 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f9400054720 con 0x7f94100b1730 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 10 client.4214.objecter ms_dispatch 0x7f9410000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 3 client.4214.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 3 client.4214.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9400050060 con 0x7f94100b1730 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:39.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.597+0000 7f940cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:39.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.792+0000 7f941646d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:39.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.792+0000 7f941646d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:39.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.792+0000 7f941646d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:20:39.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.792+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94100b5ac0 con 0x7f94100b1730 2024-03-25T19:20:39.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.818+0000 7f940cff9640 1 -- 172.21.15.60:0/3445379482 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f9400052160 con 0x7f94100b1730 2024-03-25T19:20:39.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.818+0000 7f940cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:39.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.818+0000 7f940cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:39.818 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:20:39.818 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 10 librados: watch_flush enter 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 10 librados: watch_flush exit 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 10 client.4214.objecter successfully canceled tick 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 >> v1:172.21.15.90:6800/1827638754 conn(0x7f93fc06e0c0 legacy=0x7f93fc070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 10 monclient: shutdown 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 >> v1:172.21.15.60:6790/0 conn(0x7f94100b1730 legacy=0x7f9410138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:39.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 shutdown_connections 2024-03-25T19:20:39.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 >> 172.21.15.60:0/3445379482 conn(0x7f94100617a0 msgr2=0x7f94100b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:39.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 shutdown_connections 2024-03-25T19:20:39.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 1 -- 172.21.15.60:0/3445379482 wait complete. 2024-03-25T19:20:39.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:39.820+0000 7f941646d640 1 librados: shutdown 2024-03-25T19:20:39.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:39.942 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:17.873 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.112 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:22.058 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.115 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:22.062 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.136 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:22.068 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.140 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:22.072 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.245 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:22.185 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.249 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:22.189 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.309 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:22.234 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:44.313 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:22.238 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.054 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:23.004 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.058 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:23.008 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.123 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:23.051 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.130 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:23.057 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.319 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:23.250 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.325 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:23.256 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.809 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:23.739 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:45.812 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:23.743 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:46.637 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:46.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.759+0000 7fc5c326e640 10 monclient: get_monmap_and_config 2024-03-25T19:20:46.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.759+0000 7fc5c326e640 10 monclient: build_initial_monmap 2024-03-25T19:20:46.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.759+0000 7fc5c326e640 10 monclient: monmap: 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:20:46.759869+0000 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:20:46.759869+0000 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:46.759 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.759+0000 7fc5c326e640 1 Processor -- start 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.760+0000 7fc5c326e640 1 -- start start 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.760+0000 7fc5c326e640 10 monclient: init 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5bc0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5bc0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5bc14be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0af940 con 0x7fc5bc0b1730 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0b0a70 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc0b0d50 con 0x7fc5bc14be90 2024-03-25T19:20:46.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:46.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c326e640 10 monclient(hunting): authenticate will time out at 6373.859300s 2024-03-25T19:20:46.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c180b640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fc5bc14be90 0x7fc5bc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:35548/0 (socket says 172.21.15.60:35548) 2024-03-25T19:20:46.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.761+0000 7fc5c180b640 1 -- 172.21.15.60:0/3531054620 learned_addr learned my addr 172.21.15.60:0/3531054620 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486070048 0 0) 0x7fc5bc0b0a70 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a8003520 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888807593 0 0) 0x7fc5bc0b0d50 con 0x7fc5bc14be90 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:46.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:20:46.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc0b0a70 con 0x7fc5bc14be90 2024-03-25T19:20:46.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528114500 0 0) 0x7fc5bc0af940 con 0x7fc5bc0b1730 2024-03-25T19:20:46.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:46.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:46.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:46.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:20:46.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.765+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc0b0d50 con 0x7fc5bc0b1730 2024-03-25T19:20:46.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4198636380 0 0) 0x7fc5a8003520 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 >> v1:172.21.15.60:6790/0 conn(0x7fc5bc14be90 legacy=0x7fc5bc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 >> v1:172.21.15.60:6789/0 conn(0x7fc5bc0b1730 legacy=0x7fc5bc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 1 monclient: found mon.noname-c 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc0b1030 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 10 monclient: _finish_auth 0 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc5b8001690 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.769+0000 7fc5b3fff640 10 monclient: dump: 2024-03-25T19:20:46.771 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.770+0000 7fc5c326e640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:20:46.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.770+0000 7fc5c326e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5b80017f0 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5b3fff640 1 -- 172.21.15.60:0/3531054620 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc5b8001c00 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5b3fff640 10 monclient: dump: 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.777 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5c326e640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.772+0000 7fc5c326e640 4 set_mon_vals no callback set 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.773+0000 7fc5c326e640 10 monclient: shutdown 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.773+0000 7fc5c326e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.773+0000 7fc5c326e640 1 -- 172.21.15.60:0/3531054620 >> v1:172.21.15.90:6789/0 conn(0x7fc5bc0b4d60 legacy=0x7fc5bc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.773+0000 7fc5c326e640 1 -- 172.21.15.60:0/3531054620 shutdown_connections 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.773+0000 7fc5c326e640 1 -- 172.21.15.60:0/3531054620 >> 172.21.15.60:0/3531054620 conn(0x7fc5bc0617a0 msgr2=0x7fc5bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:46.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.773+0000 7fc5c326e640 1 -- 172.21.15.60:0/3531054620 shutdown_connections 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.773+0000 7fc5c326e640 1 -- 172.21.15.60:0/3531054620 wait complete. 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.774+0000 7fc5c326e640 10 monclient: build_initial_monmap 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.774+0000 7fc5c326e640 10 monclient: monmap: 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.774+0000 7fc5c326e640 1 librados: starting msgr at 2024-03-25T19:20:46.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.774+0000 7fc5c326e640 1 librados: starting objecter 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 1 Processor -- start 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 1 -- start start 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 1 librados: setting wanted keys 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 1 librados: calling monclient init 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 10 monclient: init 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.775+0000 7fc5c326e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5bc0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c326e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5bc0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c326e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5bc14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c0809640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fc5bc0b4d60 0x7fc5bc142d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:35560/0 (socket says 172.21.15.60:35560) 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c0809640 1 -- 172.21.15.60:0/1120346681 learned_addr learned my addr 172.21.15.60:0/1120346681 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc141f10 con 0x7fc5bc0b1730 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc1d7ab0 con 0x7fc5bc14be90 2024-03-25T19:20:46.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5bc1d86e0 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c326e640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5c326e640 10 monclient(hunting): authenticate will time out at 6373.874629s 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5bc0b4d60 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.776+0000 7fc5b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.777+0000 7fc5b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5bc0b1730 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.777+0000 7fc5b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.777+0000 7fc5b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5bc14be90 2024-03-25T19:20:46.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.777+0000 7fc5b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:46.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.779+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958053205 0 0) 0x7fc5bc1d7ab0 con 0x7fc5bc14be90 2024-03-25T19:20:46.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.779+0000 7fc5b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:46.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.779+0000 7fc5b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:46.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.779+0000 7fc5b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:46.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.779+0000 7fc5b1ffb640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:20:46.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.779+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a0004eb0 con 0x7fc5bc14be90 2024-03-25T19:20:46.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296612340 0 0) 0x7fc5bc141f10 con 0x7fc5bc0b1730 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc1d7ab0 con 0x7fc5bc0b1730 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743235492 0 0) 0x7fc5bc1d86e0 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:20:46.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.780+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc141f10 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.786+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963817507 0 0) 0x7fc5bc1d7ab0 con 0x7fc5bc0b1730 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.786+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5bc1d86e0 con 0x7fc5bc0b1730 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.786+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc5b8001b00 con 0x7fc5bc0b1730 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.786+0000 7fc5b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.786+0000 7fc5b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.786+0000 7fc5b1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.786 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:46.787 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:46.787 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:46.787 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:46.787 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:46.787 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:46.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.786+0000 7fc5b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:46.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.787+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952163257 0 0) 0x7fc5bc141f10 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.787+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5bc1d7ab0 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.787+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc5ac0018c0 con 0x7fc5bc0b4d60 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.787+0000 7fc5b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.787+0000 7fc5b1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.787+0000 7fc5b1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:46.789 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.788+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3382600697 0 0) 0x7fc5a0004eb0 con 0x7fc5bc14be90 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.788+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5bc141f10 con 0x7fc5bc14be90 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.788+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc5b4001bd0 con 0x7fc5bc14be90 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.788+0000 7fc5b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.788+0000 7fc5b1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.788+0000 7fc5b1ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:46.790 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3133195108 0 0) 0x7fc5bc1d86e0 con 0x7fc5bc0b1730 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 >> v1:172.21.15.60:6790/0 conn(0x7fc5bc0b4d60 legacy=0x7fc5bc142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 >> v1:172.21.15.90:6789/0 conn(0x7fc5bc14be90 legacy=0x7fc5bc1e31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:46.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 1 monclient: found mon.a 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc1d8870 con 0x7fc5bc0b1730 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 5 monclient: authenticate success, global_id 4224 2024-03-25T19:20:46.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 10 monclient: _renew_subs 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc140db0 con 0x7fc5bc0b1730 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 10 monclient: _renew_subs 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5bc143640 con 0x7fc5bc0b1730 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.792+0000 7fc5c326e640 1 librados: init done 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.793+0000 7fc5c326e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:46.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.793+0000 7fc5c326e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:46.794 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.793+0000 7fc5c326e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:46.794 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.793+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5bc0627b0 con 0x7fc5bc0b1730 2024-03-25T19:20:46.795 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.796+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5b80038a0 con 0x7fc5bc0b1730 2024-03-25T19:20:46.795 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.796+0000 7fc5b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:46.796 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:46.796+0000 7fc5b37fe640 4 set_mon_vals no callback set 2024-03-25T19:20:47.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.092+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc5b8001250 con 0x7fc5bc0b1730 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.092+0000 7fc5b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.092+0000 7fc5b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.092+0000 7fc5b1ffb640 10 monclient: dump: 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:47.092 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:47.093 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:47.093 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:47.093 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:47.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.093+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fc5b8019a20 con 0x7fc5bc0b1730 2024-03-25T19:20:47.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.094+0000 7fc5c0809640 10 monclient: get_auth_request con 0x7fc5a006e0b0 auth_method 0 2024-03-25T19:20:47.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.094+0000 7fc5b1ffb640 10 client.4224.objecter ms_handle_connect 0x7fc5a006e0b0 2024-03-25T19:20:47.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.095+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fc5b80536f0 con 0x7fc5bc0b1730 2024-03-25T19:20:47.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.095+0000 7fc5b1ffb640 10 client.4224.objecter ms_dispatch 0x7fc5bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:47.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.095+0000 7fc5b1ffb640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:47.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.095+0000 7fc5b1ffb640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:47.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.095+0000 7fc5b1ffb640 20 client.4224.objecter dump_active .. 0 homeless 2024-03-25T19:20:47.104 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.105+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5b804f030 con 0x7fc5bc0b1730 2024-03-25T19:20:47.104 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.105+0000 7fc5b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:47.105 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.105+0000 7fc5b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:47.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.354+0000 7fc5c326e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:47.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.355+0000 7fc5c326e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:47.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.355+0000 7fc5c326e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:20:47.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.355+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5bc137ba0 con 0x7fc5bc0b1730 2024-03-25T19:20:47.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.370+0000 7fc5b1ffb640 1 -- 172.21.15.60:0/1120346681 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fc5b8051130 con 0x7fc5bc0b1730 2024-03-25T19:20:47.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.370+0000 7fc5b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:47.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.370+0000 7fc5b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:47.370 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:20:47.370 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:47.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 10 librados: watch_flush enter 2024-03-25T19:20:47.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 10 librados: watch_flush exit 2024-03-25T19:20:47.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 20 client.4224.objecter shutdown clearing up homeless session... 2024-03-25T19:20:47.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 10 client.4224.objecter successfully canceled tick 2024-03-25T19:20:47.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 >> v1:172.21.15.90:6800/1827638754 conn(0x7fc5a006e0b0 legacy=0x7fc5a0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 10 monclient: shutdown 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 >> v1:172.21.15.60:6789/0 conn(0x7fc5bc0b1730 legacy=0x7fc5bc1d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 shutdown_connections 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 >> 172.21.15.60:0/1120346681 conn(0x7fc5bc0617a0 msgr2=0x7fc5bc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 shutdown_connections 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.372+0000 7fc5c326e640 1 -- 172.21.15.60:0/1120346681 wait complete. 2024-03-25T19:20:47.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:47.373+0000 7fc5c326e640 1 librados: shutdown 2024-03-25T19:20:47.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:48.297 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:00:26.222 35743== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:48.481 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:00:26.413 35742== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:48.505 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:00:26.452 35740== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:48.564 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:00:26.504 35741== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:49.488 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:00:27.438 35306== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:49.840 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:00:27.770 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:49.942 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:00:27.870 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:50.433 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:00:28.363 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T19:20:54.188 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:20:54.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.334+0000 7fdbe73f4640 10 monclient: get_monmap_and_config 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.334+0000 7fdbe73f4640 10 monclient: build_initial_monmap 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.334+0000 7fdbe73f4640 10 monclient: monmap: 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:20:54.335124+0000 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:20:54.335124+0000 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:54.334 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:54.335 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:20:54.335 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:20:54.335 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.335+0000 7fdbe73f4640 1 Processor -- start 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.335+0000 7fdbe73f4640 1 -- start start 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.335+0000 7fdbe73f4640 10 monclient: init 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe00b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:54.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00af940 con 0x7fdbe00b1730 2024-03-25T19:20:54.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00b0a70 con 0x7fdbe00b4d60 2024-03-25T19:20:54.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00b0d50 con 0x7fdbe014be90 2024-03-25T19:20:54.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:54.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.336+0000 7fdbe73f4640 10 monclient(hunting): authenticate will time out at 6381.434583s 2024-03-25T19:20:54.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.337+0000 7fdbe5991640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fdbe014be90 0x7fdbe014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:35562/0 (socket says 172.21.15.60:35562) 2024-03-25T19:20:54.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.337+0000 7fdbe5991640 1 -- 172.21.15.60:0/1492046219 learned_addr learned my addr 172.21.15.60:0/1492046219 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:54.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455326162 0 0) 0x7fdbe00b0a70 con 0x7fdbe00b4d60 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbcc003520 con 0x7fdbe00b4d60 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457833770 0 0) 0x7fdbe00b0d50 con 0x7fdbe014be90 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe00b0a70 con 0x7fdbe014be90 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999307124 0 0) 0x7fdbe00af940 con 0x7fdbe00b1730 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:20:54.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:54.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:54.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:20:54.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.341+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe00b0d50 con 0x7fdbe00b1730 2024-03-25T19:20:54.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3746286256 0 0) 0x7fdbcc003520 con 0x7fdbe00b4d60 2024-03-25T19:20:54.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 >> v1:172.21.15.60:6790/0 conn(0x7fdbe014be90 legacy=0x7fdbe014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:54.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 >> v1:172.21.15.60:6789/0 conn(0x7fdbe00b1730 legacy=0x7fdbe00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:54.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:20:54.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 1 monclient: found mon.noname-c 2024-03-25T19:20:54.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00b1030 con 0x7fdbe00b4d60 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 10 monclient: _finish_auth 0 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdbdc001580 con 0x7fdbe00b4d60 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.345+0000 7fdbd7fff640 10 monclient: dump: 2024-03-25T19:20:54.347 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.346+0000 7fdbe73f4640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:20:54.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.346+0000 7fdbe73f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:20:54.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbdc0017d0 con 0x7fdbe00b4d60 2024-03-25T19:20:54.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbe73f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbe73f4640 4 set_mon_vals no callback set 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbd7fff640 1 -- 172.21.15.60:0/1492046219 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdbdc001b60 con 0x7fdbe00b4d60 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbd7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.348+0000 7fdbd7fff640 10 monclient: dump: 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:54.351 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.349+0000 7fdbe73f4640 10 monclient: shutdown 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.349+0000 7fdbe73f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.349+0000 7fdbe73f4640 1 -- 172.21.15.60:0/1492046219 >> v1:172.21.15.90:6789/0 conn(0x7fdbe00b4d60 legacy=0x7fdbe013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.349+0000 7fdbe73f4640 1 -- 172.21.15.60:0/1492046219 shutdown_connections 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.349+0000 7fdbe73f4640 1 -- 172.21.15.60:0/1492046219 >> 172.21.15.60:0/1492046219 conn(0x7fdbe00617a0 msgr2=0x7fdbe0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.349+0000 7fdbe73f4640 1 -- 172.21.15.60:0/1492046219 shutdown_connections 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.349+0000 7fdbe73f4640 1 -- 172.21.15.60:0/1492046219 wait complete. 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.350+0000 7fdbe73f4640 10 monclient: build_initial_monmap 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.350+0000 7fdbe73f4640 10 monclient: monmap: 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:20:54.352 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.351+0000 7fdbe73f4640 1 librados: starting msgr at 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.351+0000 7fdbe73f4640 1 librados: starting objecter 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.351+0000 7fdbe73f4640 1 Processor -- start 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.351+0000 7fdbe73f4640 1 -- start start 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.351+0000 7fdbe73f4640 1 librados: setting wanted keys 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.351+0000 7fdbe73f4640 1 librados: calling monclient init 2024-03-25T19:20:54.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.351+0000 7fdbe73f4640 10 monclient: init 2024-03-25T19:20:54.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe00b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe00b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe014be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe01d79d0 con 0x7fdbe00b4d60 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe0140d00 con 0x7fdbe014be90 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe01d8570 con 0x7fdbe00b1730 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.352+0000 7fdbe73f4640 10 monclient(hunting): authenticate will time out at 6381.450600s 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbe5190640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fdbe00b1730 0x7fdbe01405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:35568/0 (socket says 172.21.15.60:35568) 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbe5190640 1 -- 172.21.15.60:0/262783192 learned_addr learned my addr 172.21.15.60:0/262783192 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbd5ffb640 10 client.?.objecter ms_handle_connect 0x7fdbe00b1730 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:54.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbd5ffb640 10 client.?.objecter ms_handle_connect 0x7fdbe00b4d60 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbd5ffb640 10 client.?.objecter ms_handle_connect 0x7fdbe014be90 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.353+0000 7fdbd5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942700095 0 0) 0x7fdbe0140d00 con 0x7fdbe014be90 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc4004eb0 con 0x7fdbe014be90 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834460175 0 0) 0x7fdbe01d8570 con 0x7fdbe00b1730 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:54.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 10 monclient(hunting): my global_id is 4226 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.356+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe0140d00 con 0x7fdbe00b1730 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.357+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 150809505 0 0) 0x7fdbe01d79d0 con 0x7fdbe00b4d60 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.357+0000 7fdbd5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.357+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.357+0000 7fdbd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.357+0000 7fdbd5ffb640 10 monclient(hunting): my global_id is 4230 2024-03-25T19:20:54.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.357+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe01d8570 con 0x7fdbe00b4d60 2024-03-25T19:20:54.361 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.361+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3017035189 0 0) 0x7fdbc4004eb0 con 0x7fdbe014be90 2024-03-25T19:20:54.361 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.361+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe01d79d0 con 0x7fdbe014be90 2024-03-25T19:20:54.361 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.362+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdbd8001b50 con 0x7fdbe014be90 2024-03-25T19:20:54.361 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.362+0000 7fdbd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.362+0000 7fdbd5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.362+0000 7fdbd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.362 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.362+0000 7fdbd5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:20:54.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.364+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 981270494 0 0) 0x7fdbe01d8570 con 0x7fdbe00b4d60 2024-03-25T19:20:54.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.364+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc4004eb0 con 0x7fdbe00b4d60 2024-03-25T19:20:54.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.364+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdbd00018f0 con 0x7fdbe00b4d60 2024-03-25T19:20:54.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.364+0000 7fdbd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:54.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.364+0000 7fdbd5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:20:54.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.364+0000 7fdbd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.365+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1719921065 0 0) 0x7fdbe0140d00 con 0x7fdbe00b1730 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.365+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe01d8570 con 0x7fdbe00b1730 2024-03-25T19:20:54.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.365+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdbdc001ae0 con 0x7fdbe00b1730 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.365+0000 7fdbd5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.365+0000 7fdbd5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.365+0000 7fdbd5ffb640 10 monclient(hunting): dump: 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:54.366 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2619001913 0 0) 0x7fdbe01d79d0 con 0x7fdbe014be90 2024-03-25T19:20:54.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 >> v1:172.21.15.60:6790/0 conn(0x7fdbe00b1730 legacy=0x7fdbe01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 >> v1:172.21.15.60:6789/0 conn(0x7fdbe00b4d60 legacy=0x7fdbe01d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 1 monclient: found mon.b 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe0141d50 con 0x7fdbe014be90 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 10 monclient: _renew_subs 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe013b140 con 0x7fdbe014be90 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:20:54.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 10 monclient: _renew_subs 2024-03-25T19:20:54.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:54.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe0143310 con 0x7fdbe014be90 2024-03-25T19:20:54.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.366+0000 7fdbe73f4640 1 librados: init done 2024-03-25T19:20:54.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.368+0000 7fdbe73f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:54.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.368+0000 7fdbe73f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:54.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.368+0000 7fdbe73f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:54.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.368+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe01d7730 con 0x7fdbe014be90 2024-03-25T19:20:54.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.374+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd8002020 con 0x7fdbe014be90 2024-03-25T19:20:54.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.374+0000 7fdbd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:20:54.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.374+0000 7fdbd77fe640 4 set_mon_vals no callback set 2024-03-25T19:20:54.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.389+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdbd8004d00 con 0x7fdbe014be90 2024-03-25T19:20:54.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.389+0000 7fdbd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:20:54.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.389+0000 7fdbd5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:20:54.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.389+0000 7fdbd5ffb640 10 monclient: dump: 2024-03-25T19:20:54.389 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:20:54.389 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:20:54.389 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.390 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:20:54.390 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:20:54.390 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:20:54.390 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:20:54.390 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:20:54.390 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:20:54.390 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:20:54.391 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.390+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fdbd801acd0 con 0x7fdbe014be90 2024-03-25T19:20:54.391 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.391+0000 7fdbe5190640 10 monclient: get_auth_request con 0x7fdbc406e0c0 auth_method 0 2024-03-25T19:20:54.391 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.391+0000 7fdbd5ffb640 10 client.4216.objecter ms_handle_connect 0x7fdbc406e0c0 2024-03-25T19:20:54.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.394+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fdbd8054980 con 0x7fdbe014be90 2024-03-25T19:20:54.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.394+0000 7fdbd5ffb640 10 client.4216.objecter ms_dispatch 0x7fdbe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:20:54.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.394+0000 7fdbd5ffb640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:20:54.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.394+0000 7fdbd5ffb640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:20:54.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.394+0000 7fdbd5ffb640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T19:20:54.633 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.633+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbd80502c0 con 0x7fdbe014be90 2024-03-25T19:20:54.633 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.633+0000 7fdbd5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:20:54.633 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:54.633+0000 7fdbd5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:20:55.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.017+0000 7fdbe73f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:55.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.017+0000 7fdbe73f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:55.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.017+0000 7fdbe73f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:20:55.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.017+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbe00627b0 con 0x7fdbe014be90 2024-03-25T19:20:55.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.074+0000 7fdbd5ffb640 1 -- 172.21.15.60:0/262783192 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fdbd80523c0 con 0x7fdbe014be90 2024-03-25T19:20:55.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.074+0000 7fdbd5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:20:55.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.074+0000 7fdbd5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:20:55.075 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:20:55.075 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:20:55.077 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.077+0000 7fdbe73f4640 10 librados: watch_flush enter 2024-03-25T19:20:55.077 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.077+0000 7fdbe73f4640 10 librados: watch_flush exit 2024-03-25T19:20:55.077 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.077+0000 7fdbe73f4640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T19:20:55.077 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.077+0000 7fdbe73f4640 10 client.4216.objecter successfully canceled tick 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.077+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 >> v1:172.21.15.90:6800/1827638754 conn(0x7fdbc406e0c0 legacy=0x7fdbc4070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 10 monclient: shutdown 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 >> v1:172.21.15.90:6789/0 conn(0x7fdbe014be90 legacy=0x7fdbe0142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 shutdown_connections 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 >> 172.21.15.60:0/262783192 conn(0x7fdbe00617a0 msgr2=0x7fdbe0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 shutdown_connections 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 1 -- 172.21.15.60:0/262783192 wait complete. 2024-03-25T19:20:55.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:20:55.078+0000 7fdbe73f4640 1 librados: shutdown 2024-03-25T19:20:55.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:20:56.854 INFO:tasks.ceph.osd.6.smithi090.stderr:2024-03-25T19:20:56.851+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:20:57.110 INFO:tasks.ceph.osd.7.smithi090.stderr:2024-03-25T19:20:57.100+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:20:57.194 INFO:tasks.ceph.osd.5.smithi090.stderr:2024-03-25T19:20:57.191+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:20:57.550 INFO:tasks.ceph.osd.4.smithi090.stderr:2024-03-25T19:20:57.547+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:20:58.266 INFO:tasks.ceph.osd.1.smithi060.stderr:2024-03-25T19:20:58.263+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:20:58.916 INFO:tasks.ceph.osd.2.smithi060.stderr:2024-03-25T19:20:58.869+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:20:58.945 INFO:tasks.ceph.osd.3.smithi060.stderr:2024-03-25T19:20:58.944+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:20:58.973 INFO:tasks.ceph.osd.0.smithi060.stderr:2024-03-25T19:20:58.970+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:21:01.898 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.016+0000 7fc0c538e640 10 monclient: get_monmap_and_config 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.016+0000 7fc0c538e640 10 monclient: build_initial_monmap 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.016+0000 7fc0c538e640 10 monclient: monmap: 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:02.017280+0000 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:02.017280+0000 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:02.016 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:02.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.017+0000 7fc0c538e640 1 Processor -- start 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.017+0000 7fc0c538e640 1 -- start start 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.017+0000 7fc0c538e640 10 monclient: init 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.017+0000 7fc0c538e640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.017+0000 7fc0c538e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.017+0000 7fc0c538e640 10 monclient(hunting): picked mon.noname-a con 0x7fc0c00b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.018+0000 7fc0c538e640 10 monclient(hunting): picked mon.noname-b con 0x7fc0c00b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.018+0000 7fc0c538e640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c014be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.018+0000 7fc0c538e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00af940 con 0x7fc0c00b4d60 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.018+0000 7fc0c538e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00b0a70 con 0x7fc0c014be90 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.018+0000 7fc0c538e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c00b0d50 con 0x7fc0c00b1730 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.018+0000 7fc0c538e640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:02.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.018+0000 7fc0c538e640 10 monclient(hunting): authenticate will time out at 6389.115938s 2024-03-25T19:21:02.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.025+0000 7fc0beffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc0c00b4d60 0x7fc0c013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37792/0 (socket says 172.21.15.60:37792) 2024-03-25T19:21:02.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.025+0000 7fc0beffd640 1 -- 172.21.15.60:0/799707624 learned_addr learned my addr 172.21.15.60:0/799707624 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:02.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.038+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573715798 0 0) 0x7fc0c00af940 con 0x7fc0c00b4d60 2024-03-25T19:21:02.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.038+0000 7fc0be7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:21:02.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.038+0000 7fc0be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:02.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.038+0000 7fc0be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:02.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.038+0000 7fc0be7fc640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:21:02.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.038+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ac003520 con 0x7fc0c00b4d60 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 646101563 0 0) 0x7fc0ac003520 con 0x7fc0c00b4d60 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 >> v1:172.21.15.60:6790/0 conn(0x7fc0c00b1730 legacy=0x7fc0c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 >> v1:172.21.15.90:6789/0 conn(0x7fc0c014be90 legacy=0x7fc0c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 1 monclient: found mon.noname-a 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c00b1030 con 0x7fc0c00b4d60 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 10 monclient: _finish_auth 0 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc0b0001690 con 0x7fc0c00b4d60 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:02.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0be7fc640 10 monclient: dump: 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0c538e640 5 monclient: authenticate success, global_id 4257 2024-03-25T19:21:02.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.047+0000 7fc0c538e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:02.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b00017f0 con 0x7fc0c00b4d60 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0be7fc640 1 -- 172.21.15.60:0/799707624 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc0b0001c00 con 0x7fc0c00b4d60 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0c538e640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0c538e640 4 set_mon_vals no callback set 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.056+0000 7fc0be7fc640 10 monclient: dump: 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:02.056 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:02.057 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:02.057 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:02.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.057+0000 7fc0c538e640 10 monclient: shutdown 2024-03-25T19:21:02.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.057+0000 7fc0c538e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:02.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.057+0000 7fc0c538e640 1 -- 172.21.15.60:0/799707624 >> v1:172.21.15.60:6789/0 conn(0x7fc0c00b4d60 legacy=0x7fc0c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:02.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.057+0000 7fc0c538e640 1 -- 172.21.15.60:0/799707624 shutdown_connections 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.057+0000 7fc0c538e640 1 -- 172.21.15.60:0/799707624 >> 172.21.15.60:0/799707624 conn(0x7fc0c00617a0 msgr2=0x7fc0c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.057+0000 7fc0c538e640 1 -- 172.21.15.60:0/799707624 shutdown_connections 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.057+0000 7fc0c538e640 1 -- 172.21.15.60:0/799707624 wait complete. 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 10 monclient: build_initial_monmap 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 10 monclient: monmap: 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:02.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 1 librados: starting msgr at 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 1 librados: starting objecter 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 1 Processor -- start 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 1 -- start start 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 1 librados: setting wanted keys 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 1 librados: calling monclient init 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.058+0000 7fc0c538e640 10 monclient: init 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 10 monclient(hunting): picked mon.noname-b con 0x7fc0c00b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 10 monclient(hunting): picked mon.noname-a con 0x7fc0c00b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 10 monclient(hunting): picked mon.noname-c con 0x7fc0c014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0145930 con 0x7fc0c00b4d60 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c01446c0 con 0x7fc0c00b1730 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c01449a0 con 0x7fc0c014be90 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:02.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.059+0000 7fc0c538e640 10 monclient(hunting): authenticate will time out at 6389.157154s 2024-03-25T19:21:02.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.069+0000 7fc0b7fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc0c00b4d60 0x7fc0c0136180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37838/0 (socket says 172.21.15.60:37838) 2024-03-25T19:21:02.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.069+0000 7fc0b7fff640 1 -- 172.21.15.60:0/2825809270 learned_addr learned my addr 172.21.15.60:0/2825809270 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:02.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.070+0000 7fc0b77fe640 10 client.?.objecter ms_handle_connect 0x7fc0c00b4d60 2024-03-25T19:21:02.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.070+0000 7fc0b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:02.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.073+0000 7fc0b77fe640 10 client.?.objecter ms_handle_connect 0x7fc0c00b1730 2024-03-25T19:21:02.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.073+0000 7fc0b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:02.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.078+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032020670 0 0) 0x7fc0c01446c0 con 0x7fc0c00b1730 2024-03-25T19:21:02.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.078+0000 7fc0b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:02.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.078+0000 7fc0b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:02.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.078+0000 7fc0b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:02.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.078+0000 7fc0b77fe640 10 monclient(hunting): my global_id is 4246 2024-03-25T19:21:02.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.078+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a0004eb0 con 0x7fc0c00b1730 2024-03-25T19:21:02.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.079+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233419214 0 0) 0x7fc0c0145930 con 0x7fc0c00b4d60 2024-03-25T19:21:02.079 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.079+0000 7fc0b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:02.079 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.079+0000 7fc0b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:02.079 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.079+0000 7fc0b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:02.079 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.079+0000 7fc0b77fe640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:21:02.079 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.079+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c01446c0 con 0x7fc0c00b4d60 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.090+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2094862864 0 0) 0x7fc0a0004eb0 con 0x7fc0c00b1730 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.090+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0c0145930 con 0x7fc0c00b1730 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.090+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc0b0001bc0 con 0x7fc0c00b1730 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.090+0000 7fc0b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.090+0000 7fc0b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.090+0000 7fc0b77fe640 10 monclient(hunting): dump: 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:02.090 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:02.091 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:02.091 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:02.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.090+0000 7fc0b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2544542921 0 0) 0x7fc0c0145930 con 0x7fc0c00b1730 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 >> v1:172.21.15.60:6790/0 conn(0x7fc0c014be90 legacy=0x7fc0c01e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 >> v1:172.21.15.60:6789/0 conn(0x7fc0c00b4d60 legacy=0x7fc0c0136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 1 monclient: found mon.b 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0144c80 con 0x7fc0c00b1730 2024-03-25T19:21:02.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 5 monclient: authenticate success, global_id 4246 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 10 monclient: _renew_subs 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c013b980 con 0x7fc0c00b1730 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 10 monclient: _renew_subs 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c013be80 con 0x7fc0c00b1730 2024-03-25T19:21:02.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.095+0000 7fc0c538e640 1 librados: init done 2024-03-25T19:21:02.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.096+0000 7fc0c538e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:02.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.096+0000 7fc0c538e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:02.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.096+0000 7fc0c538e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:02.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.096+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c01401f0 con 0x7fc0c00b1730 2024-03-25T19:21:02.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.393+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0b0004a90 con 0x7fc0c00b1730 2024-03-25T19:21:02.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.393+0000 7fc0b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:02.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.393+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc0b0001250 con 0x7fc0c00b1730 2024-03-25T19:21:02.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.393+0000 7fc0b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:02.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.393+0000 7fc0b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.393+0000 7fc0b77fe640 10 monclient: dump: 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:02.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.393+0000 7fc0bdffb640 4 set_mon_vals no callback set 2024-03-25T19:21:02.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.394+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fc0b001ac90 con 0x7fc0c00b1730 2024-03-25T19:21:02.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.395+0000 7fc0b7fff640 10 monclient: get_auth_request con 0x7fc0a006dcb0 auth_method 0 2024-03-25T19:21:02.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.395+0000 7fc0b77fe640 10 client.4246.objecter ms_handle_connect 0x7fc0a006dcb0 2024-03-25T19:21:02.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fc0b00548e0 con 0x7fc0c00b1730 2024-03-25T19:21:02.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 10 client.4246.objecter ms_dispatch 0x7fc0c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:02.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:02.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:02.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-25T19:21:02.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0b0050220 con 0x7fc0c00b1730 2024-03-25T19:21:02.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:02.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.401+0000 7fc0b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:02.630 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.630+0000 7fc0c538e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:02.630 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.630+0000 7fc0c538e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:02.630 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.630+0000 7fc0c538e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:02.630 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.630+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0c0145680 con 0x7fc0c00b1730 2024-03-25T19:21:02.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.692+0000 7fc0b77fe640 1 -- 172.21.15.60:0/2825809270 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fc0b0052320 con 0x7fc0c00b1730 2024-03-25T19:21:02.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.692+0000 7fc0b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:02.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.692+0000 7fc0b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:02.692 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:02.692 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:02.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.694+0000 7fc0c538e640 10 librados: watch_flush enter 2024-03-25T19:21:02.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.694+0000 7fc0c538e640 10 librados: watch_flush exit 2024-03-25T19:21:02.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.694+0000 7fc0c538e640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-25T19:21:02.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.694+0000 7fc0c538e640 10 client.4246.objecter successfully canceled tick 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.694+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 >> v1:172.21.15.90:6800/1827638754 conn(0x7fc0a006dcb0 legacy=0x7fc0a0070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.694+0000 7fc0c538e640 10 monclient: shutdown 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.695+0000 7fc0c538e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.695+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 >> v1:172.21.15.90:6789/0 conn(0x7fc0c00b1730 legacy=0x7fc0c013b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.695+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 shutdown_connections 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.695+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 >> 172.21.15.60:0/2825809270 conn(0x7fc0c00617a0 msgr2=0x7fc0c00b6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.695+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 shutdown_connections 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.695+0000 7fc0c538e640 1 -- 172.21.15.60:0/2825809270 wait complete. 2024-03-25T19:21:02.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:02.695+0000 7fc0c538e640 1 librados: shutdown 2024-03-25T19:21:02.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:09.513 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:09.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.616+0000 7ff606d8f640 10 monclient: get_monmap_and_config 2024-03-25T19:21:09.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.616+0000 7ff606d8f640 10 monclient: build_initial_monmap 2024-03-25T19:21:09.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.616+0000 7ff606d8f640 10 monclient: monmap: 2024-03-25T19:21:09.615 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:09.615 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:09.615 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:09.616790+0000 2024-03-25T19:21:09.615 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:09.616790+0000 2024-03-25T19:21:09.616 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:09.616 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:09.616 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:09.616 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:09.616 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:09.616 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.616+0000 7ff606d8f640 1 Processor -- start 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.616+0000 7ff606d8f640 1 -- start start 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.616+0000 7ff606d8f640 10 monclient: init 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6000b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 10 monclient(hunting): picked mon.noname-c con 0x7ff6000b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 10 monclient(hunting): picked mon.noname-a con 0x7ff60014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:09.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6000af940 con 0x7ff60014be90 2024-03-25T19:21:09.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6000b0a70 con 0x7ff6000b1730 2024-03-25T19:21:09.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6000b0d50 con 0x7ff6000b4d60 2024-03-25T19:21:09.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:09.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.617+0000 7ff606d8f640 10 monclient(hunting): authenticate will time out at 6396.715395s 2024-03-25T19:21:09.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.628+0000 7ff5fffff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff6000b1730 0x7ff6000b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:55148/0 (socket says 172.21.15.60:55148) 2024-03-25T19:21:09.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.628+0000 7ff5fffff640 1 -- 172.21.15.60:0/3535566329 learned_addr learned my addr 172.21.15.60:0/3535566329 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:09.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.644+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063724900 0 0) 0x7ff6000b0a70 con 0x7ff6000b1730 2024-03-25T19:21:09.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.644+0000 7ff5ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:09.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.644+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:09.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.644+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:09.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.644+0000 7ff5ff7fe640 10 monclient(hunting): my global_id is 4249 2024-03-25T19:21:09.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.644+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f0003520 con 0x7ff6000b1730 2024-03-25T19:21:09.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.647+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653656568 0 0) 0x7ff6000af940 con 0x7ff60014be90 2024-03-25T19:21:09.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.647+0000 7ff5ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:09.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.647+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:09.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.647+0000 7ff5ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:09.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.647+0000 7ff5ff7fe640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:21:09.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.647+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6000b0a70 con 0x7ff60014be90 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3210454659 0 0) 0x7ff6000b0a70 con 0x7ff60014be90 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 >> v1:172.21.15.60:6790/0 conn(0x7ff6000b4d60 legacy=0x7ff60013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 >> v1:172.21.15.90:6789/0 conn(0x7ff6000b1730 legacy=0x7ff6000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 1 monclient: found mon.noname-a 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000b1030 con 0x7ff60014be90 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:09.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff5f80019f0 con 0x7ff60014be90 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff5ff7fe640 10 monclient: dump: 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:09.655 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:09.656 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:09.656 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:09.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff606d8f640 5 monclient: authenticate success, global_id 4263 2024-03-25T19:21:09.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.654+0000 7ff606d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5f8001b50 con 0x7ff60014be90 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff5ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff5ff7fe640 1 -- 172.21.15.60:0/3535566329 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff5f8001fa0 con 0x7ff60014be90 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff606d8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff606d8f640 4 set_mon_vals no callback set 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff5ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff5ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.660+0000 7ff5ff7fe640 10 monclient: dump: 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.660 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:09.661 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:09.661 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:09.661 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:09.661 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:09.661 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:09.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.661+0000 7ff606d8f640 10 monclient: shutdown 2024-03-25T19:21:09.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.661+0000 7ff606d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:09.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.661+0000 7ff606d8f640 1 -- 172.21.15.60:0/3535566329 >> v1:172.21.15.60:6789/0 conn(0x7ff60014be90 legacy=0x7ff60014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:09.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.661+0000 7ff606d8f640 1 -- 172.21.15.60:0/3535566329 shutdown_connections 2024-03-25T19:21:09.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.661+0000 7ff606d8f640 1 -- 172.21.15.60:0/3535566329 >> 172.21.15.60:0/3535566329 conn(0x7ff6000617a0 msgr2=0x7ff600061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:09.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.661+0000 7ff606d8f640 1 -- 172.21.15.60:0/3535566329 shutdown_connections 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.661+0000 7ff606d8f640 1 -- 172.21.15.60:0/3535566329 wait complete. 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 10 monclient: build_initial_monmap 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 10 monclient: monmap: 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:09.663 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 1 librados: starting msgr at 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 1 librados: starting objecter 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 1 Processor -- start 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 1 -- start start 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 1 librados: setting wanted keys 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 1 librados: calling monclient init 2024-03-25T19:21:09.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.662+0000 7ff606d8f640 10 monclient: init 2024-03-25T19:21:09.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:09.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:09.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 10 monclient(hunting): picked mon.noname-b con 0x7ff6000b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:09.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 10 monclient(hunting): picked mon.noname-a con 0x7ff6000b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:09.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 10 monclient(hunting): picked mon.noname-c con 0x7ff60014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:09.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff600142d00 con 0x7ff6000b4d60 2024-03-25T19:21:09.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff600140380 con 0x7ff6000b1730 2024-03-25T19:21:09.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6001d74b0 con 0x7ff60014be90 2024-03-25T19:21:09.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:09.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.663+0000 7ff606d8f640 10 monclient(hunting): authenticate will time out at 6396.761651s 2024-03-25T19:21:09.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.678+0000 7ff5fffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff6000b4d60 0x7ff600132e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59822/0 (socket says 172.21.15.60:59822) 2024-03-25T19:21:09.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.678+0000 7ff5fffff640 1 -- 172.21.15.60:0/3933923791 learned_addr learned my addr 172.21.15.60:0/3933923791 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:09.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.680+0000 7ff5fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff6000b4d60 2024-03-25T19:21:09.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.680+0000 7ff5fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:09.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.683+0000 7ff5fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff6000b1730 2024-03-25T19:21:09.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.683+0000 7ff5fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:09.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.694+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439328467 0 0) 0x7ff600142d00 con 0x7ff6000b4d60 2024-03-25T19:21:09.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.694+0000 7ff5fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:09.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.694+0000 7ff5fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:09.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.694+0000 7ff5fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:09.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.694+0000 7ff5fd7fa640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:21:09.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.694+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e4004eb0 con 0x7ff6000b4d60 2024-03-25T19:21:09.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.695+0000 7ff5fd7fa640 10 client.?.objecter ms_handle_connect 0x7ff60014be90 2024-03-25T19:21:09.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.695+0000 7ff5fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:09.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.700+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307901904 0 0) 0x7ff600140380 con 0x7ff6000b1730 2024-03-25T19:21:09.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.700+0000 7ff5fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:09.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.700+0000 7ff5fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:09.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.700+0000 7ff5fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:09.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.700+0000 7ff5fd7fa640 10 monclient(hunting): my global_id is 4252 2024-03-25T19:21:09.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.700+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff600142d00 con 0x7ff6000b1730 2024-03-25T19:21:09.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.709+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841787762 0 0) 0x7ff6001d74b0 con 0x7ff60014be90 2024-03-25T19:21:09.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.709+0000 7ff5fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:09.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.709+0000 7ff5fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:09.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.709+0000 7ff5fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:09.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.709+0000 7ff5fd7fa640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:21:09.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.709+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff600140380 con 0x7ff60014be90 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.716+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3444699683 0 0) 0x7ff600142d00 con 0x7ff6000b1730 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.716+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6001d74b0 con 0x7ff6000b1730 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.716+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff5f4001510 con 0x7ff6000b1730 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.716+0000 7ff5fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.716+0000 7ff5fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.716+0000 7ff5fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.716 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.717 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:09.717 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:09.717 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:09.717 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:09.717 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:09.717 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:09.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.716+0000 7ff5fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:09.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.730+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900147647 0 0) 0x7ff5e4004eb0 con 0x7ff6000b4d60 2024-03-25T19:21:09.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.730+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff600142d00 con 0x7ff6000b4d60 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.730+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff5ec001990 con 0x7ff6000b4d60 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.730+0000 7ff5fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.730+0000 7ff5fd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.730+0000 7ff5fd7fa640 10 monclient(hunting): dump: 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:09.731 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.730+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 961462776 0 0) 0x7ff6001d74b0 con 0x7ff6000b1730 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 >> v1:172.21.15.60:6790/0 conn(0x7ff60014be90 legacy=0x7ff600141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 >> v1:172.21.15.60:6789/0 conn(0x7ff6000b4d60 legacy=0x7ff600132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 1 monclient: found mon.b 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff600143900 con 0x7ff6000b1730 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff5fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 5 monclient: authenticate success, global_id 4252 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 10 monclient: _renew_subs 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:09.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6001d80e0 con 0x7ff6000b1730 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 10 monclient: _renew_subs 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff600133710 con 0x7ff6000b1730 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.731+0000 7ff606d8f640 1 librados: init done 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.732+0000 7ff606d8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.732+0000 7ff606d8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.732+0000 7ff606d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:09.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:09.732+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff600143a90 con 0x7ff6000b1730 2024-03-25T19:21:10.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.023+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5f4001a00 con 0x7ff6000b1730 2024-03-25T19:21:10.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.023+0000 7ff5fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:10.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.023+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff5f4001bf0 con 0x7ff6000b1730 2024-03-25T19:21:10.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.023+0000 7ff5fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:10.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.023+0000 7ff5fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:10.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.023+0000 7ff5fd7fa640 10 monclient: dump: 2024-03-25T19:21:10.024 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.023+0000 7ff5feffd640 4 set_mon_vals no callback set 2024-03-25T19:21:10.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.024+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7ff5f4019780 con 0x7ff6000b1730 2024-03-25T19:21:10.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.025+0000 7ff5fffff640 10 monclient: get_auth_request con 0x7ff5e406df20 auth_method 0 2024-03-25T19:21:10.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.025+0000 7ff5fd7fa640 10 client.4252.objecter ms_handle_connect 0x7ff5e406df20 2024-03-25T19:21:10.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.028+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7ff5f40533d0 con 0x7ff6000b1730 2024-03-25T19:21:10.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.028+0000 7ff5fd7fa640 10 client.4252.objecter ms_dispatch 0x7ff600000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:10.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.028+0000 7ff5fd7fa640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:10.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.028+0000 7ff5fd7fa640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:10.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.028+0000 7ff5fd7fa640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T19:21:10.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.029+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5f404eee0 con 0x7ff6000b1730 2024-03-25T19:21:10.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.029+0000 7ff5fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:10.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.029+0000 7ff5fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:10.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.289+0000 7ff606d8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:10.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.289+0000 7ff606d8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:10.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.289+0000 7ff606d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:10.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.289+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6001d7640 con 0x7ff6000b1730 2024-03-25T19:21:10.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.308+0000 7ff5fd7fa640 1 -- 172.21.15.60:0/3933923791 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7ff5f4050e10 con 0x7ff6000b1730 2024-03-25T19:21:10.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.308+0000 7ff5fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:10.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.308+0000 7ff5fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:10.308 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:10.308 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 10 librados: watch_flush enter 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 10 librados: watch_flush exit 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 10 client.4252.objecter successfully canceled tick 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 >> v1:172.21.15.90:6800/1827638754 conn(0x7ff5e406df20 legacy=0x7ff5e40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 10 monclient: shutdown 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 >> v1:172.21.15.90:6789/0 conn(0x7ff6000b1730 legacy=0x7ff6001d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 shutdown_connections 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 >> 172.21.15.60:0/3933923791 conn(0x7ff6000617a0 msgr2=0x7ff6000b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:10.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 shutdown_connections 2024-03-25T19:21:10.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 1 -- 172.21.15.60:0/3933923791 wait complete. 2024-03-25T19:21:10.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:10.311+0000 7ff606d8f640 1 librados: shutdown 2024-03-25T19:21:10.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:13.777 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:14.489 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:13.834 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:14.546 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:13.974 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:14.685 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:13.990 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:14.644 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:14.035 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:14.746 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:14.051 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:14.705 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:14.073 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:14.784 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:14.273 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:14.926 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:14.279 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:14.991 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:17.128 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.246+0000 7f79990cf640 10 monclient: get_monmap_and_config 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.246+0000 7f79990cf640 10 monclient: build_initial_monmap 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.246+0000 7f79990cf640 10 monclient: monmap: 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:17.247181+0000 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:17.247181+0000 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:17.246 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:17.247 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:17.247 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 1 Processor -- start 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 1 -- start start 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient: init 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient(hunting): picked mon.noname-a con 0x7f79940a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient(hunting): picked mon.noname-b con 0x7f799413f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient(hunting): picked mon.noname-c con 0x7f79941430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7994142520 con 0x7f79940a98f0 2024-03-25T19:21:17.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79940b28a0 con 0x7f79941430c0 2024-03-25T19:21:17.249 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79940b2a30 con 0x7f799413f710 2024-03-25T19:21:17.249 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:17.249 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.247+0000 7f79990cf640 10 monclient(hunting): authenticate will time out at 6404.345759s 2024-03-25T19:21:17.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.253+0000 7f799359e640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f79941430c0 0x7f799414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:42544/0 (socket says 172.21.15.60:42544) 2024-03-25T19:21:17.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.253+0000 7f799359e640 1 -- 172.21.15.60:0/1675908005 learned_addr learned my addr 172.21.15.60:0/1675908005 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:17.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.268+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812823302 0 0) 0x7f79940b28a0 con 0x7f79941430c0 2024-03-25T19:21:17.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.268+0000 7f799259c640 10 monclient(hunting): _negotiate 2024-03-25T19:21:17.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.268+0000 7f799259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:17.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.268+0000 7f799259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:17.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.268+0000 7f799259c640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:21:17.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.268+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7980003520 con 0x7f79941430c0 2024-03-25T19:21:17.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.271+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502894359 0 0) 0x7f79940b2a30 con 0x7f799413f710 2024-03-25T19:21:17.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.271+0000 7f799259c640 10 monclient(hunting): _negotiate 2024-03-25T19:21:17.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.271+0000 7f799259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:17.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.271+0000 7f799259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:17.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.271+0000 7f799259c640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:21:17.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.271+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79940b28a0 con 0x7f799413f710 2024-03-25T19:21:17.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.272+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018924335 0 0) 0x7f7994142520 con 0x7f79940a98f0 2024-03-25T19:21:17.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.272+0000 7f799259c640 10 monclient(hunting): _negotiate 2024-03-25T19:21:17.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.272+0000 7f799259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:17.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.272+0000 7f799259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:17.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.272+0000 7f799259c640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:21:17.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.272+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79940b2a30 con 0x7f79940a98f0 2024-03-25T19:21:17.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2367530801 0 0) 0x7f79940b28a0 con 0x7f799413f710 2024-03-25T19:21:17.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 >> v1:172.21.15.90:6789/0 conn(0x7f79941430c0 legacy=0x7f799414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:17.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 >> v1:172.21.15.60:6789/0 conn(0x7f79940a98f0 legacy=0x7f79940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:17.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 10 monclient: _finish_hunting 0 2024-03-25T19:21:17.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 1 monclient: found mon.noname-b 2024-03-25T19:21:17.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:17.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79940b44c0 con 0x7f799413f710 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 10 monclient: _finish_auth 0 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 10 monclient: _check_auth_tickets 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f797c001670 con 0x7f799413f710 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f799259c640 10 monclient: dump: 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:17.293 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:17.294 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:17.294 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:17.294 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:17.294 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:17.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f79990cf640 5 monclient: authenticate success, global_id 4253 2024-03-25T19:21:17.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.292+0000 7f79990cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:17.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f797c0017d0 con 0x7f799413f710 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f799259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f799259c640 1 -- 172.21.15.60:0/1675908005 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f797c001be0 con 0x7f799413f710 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f799259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f799259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f799259c640 10 monclient: dump: 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:17.305 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:17.306 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:17.306 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:17.306 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:17.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f79990cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:17.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.305+0000 7f79990cf640 4 set_mon_vals no callback set 2024-03-25T19:21:17.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.306+0000 7f79990cf640 10 monclient: shutdown 2024-03-25T19:21:17.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.306+0000 7f79990cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:17.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.306+0000 7f79990cf640 1 -- 172.21.15.60:0/1675908005 >> v1:172.21.15.60:6790/0 conn(0x7f799413f710 legacy=0x7f79941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:17.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.306+0000 7f79990cf640 1 -- 172.21.15.60:0/1675908005 shutdown_connections 2024-03-25T19:21:17.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.306+0000 7f79990cf640 1 -- 172.21.15.60:0/1675908005 >> 172.21.15.60:0/1675908005 conn(0x7f79940b7ea0 msgr2=0x7f79940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:17.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.306+0000 7f79990cf640 1 -- 172.21.15.60:0/1675908005 shutdown_connections 2024-03-25T19:21:17.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.306+0000 7f79990cf640 1 -- 172.21.15.60:0/1675908005 wait complete. 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.308+0000 7f79990cf640 10 monclient: build_initial_monmap 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.308+0000 7f79990cf640 10 monclient: monmap: 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:17.308 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.308+0000 7f79990cf640 1 librados: starting msgr at 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.308+0000 7f79990cf640 1 librados: starting objecter 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.308+0000 7f79990cf640 1 Processor -- start 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.309+0000 7f79990cf640 1 -- start start 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.309+0000 7f79990cf640 1 librados: setting wanted keys 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.309+0000 7f79990cf640 1 librados: calling monclient init 2024-03-25T19:21:17.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.309+0000 7f79990cf640 10 monclient: init 2024-03-25T19:21:17.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.309+0000 7f79990cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:17.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.309+0000 7f79990cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:17.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.309+0000 7f79990cf640 10 monclient(hunting): picked mon.noname-b con 0x7f79940a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:17.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.310+0000 7f79990cf640 10 monclient(hunting): picked mon.noname-a con 0x7f799413f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:17.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.310+0000 7f79990cf640 10 monclient(hunting): picked mon.noname-c con 0x7f79941430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:17.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.310+0000 7f79990cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799413bfc0 con 0x7f799413f710 2024-03-25T19:21:17.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.310+0000 7f79990cf640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79941369f0 con 0x7f79940a98f0 2024-03-25T19:21:17.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.310+0000 7f79990cf640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7994136cd0 con 0x7f79941430c0 2024-03-25T19:21:17.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.310+0000 7f79990cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:17.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.310+0000 7f79990cf640 10 monclient(hunting): authenticate will time out at 6404.408036s 2024-03-25T19:21:17.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.321+0000 7f798a59c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f799413f710 0x7f79941df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:43594/0 (socket says 172.21.15.60:43594) 2024-03-25T19:21:17.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.321+0000 7f798a59c640 1 -- 172.21.15.60:0/1665592224 learned_addr learned my addr 172.21.15.60:0/1665592224 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:17.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.321+0000 7f798bfff640 10 client.?.objecter ms_handle_connect 0x7f799413f710 2024-03-25T19:21:17.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.321+0000 7f798bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:17.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.327+0000 7f798bfff640 10 client.?.objecter ms_handle_connect 0x7f79940a98f0 2024-03-25T19:21:17.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.327+0000 7f798bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:17.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.329+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043877795 0 0) 0x7f799413bfc0 con 0x7f799413f710 2024-03-25T19:21:17.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.329+0000 7f798bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:17.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.329+0000 7f798bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:17.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.329+0000 7f798bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:17.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.329+0000 7f798bfff640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:21:17.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.329+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7974004eb0 con 0x7f799413f710 2024-03-25T19:21:17.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.336+0000 7f798bfff640 10 client.?.objecter ms_handle_connect 0x7f79941430c0 2024-03-25T19:21:17.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.336+0000 7f798bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:17.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.338+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325128970 0 0) 0x7f79941369f0 con 0x7f79940a98f0 2024-03-25T19:21:17.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.338+0000 7f798bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:17.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.338+0000 7f798bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:17.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.338+0000 7f798bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:17.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.338+0000 7f798bfff640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:21:17.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.338+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799413bfc0 con 0x7f79940a98f0 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.346+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 520797453 0 0) 0x7f7974004eb0 con 0x7f799413f710 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.346+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79941369f0 con 0x7f799413f710 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.346+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f797c001670 con 0x7f799413f710 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.346+0000 7f798bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.346+0000 7f798bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.346+0000 7f798bfff640 10 monclient(hunting): dump: 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:17.346 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:17.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.346+0000 7f798bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.348+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3995104841 0 0) 0x7f799413bfc0 con 0x7f79940a98f0 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.348+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7974004eb0 con 0x7f79940a98f0 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.348+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7984001870 con 0x7f79940a98f0 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.348+0000 7f798bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.348+0000 7f798bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.348+0000 7f798bfff640 10 monclient(hunting): dump: 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.348 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:17.349 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:17.349 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:17.349 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:17.349 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:17.349 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1003401146 0 0) 0x7f79941369f0 con 0x7f799413f710 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 >> v1:172.21.15.60:6790/0 conn(0x7f79941430c0 legacy=0x7f79941e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 >> v1:172.21.15.90:6789/0 conn(0x7f79940a98f0 legacy=0x7f7994137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 1 monclient: found mon.a 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7994136fb0 con 0x7f799413f710 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 10 monclient: _finish_auth 0 2024-03-25T19:21:17.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f798bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 10 monclient: _renew_subs 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799413bce0 con 0x7f799413f710 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 10 monclient: _renew_subs 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79940b7740 con 0x7f799413f710 2024-03-25T19:21:17.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.357+0000 7f79990cf640 1 librados: init done 2024-03-25T19:21:17.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.358+0000 7f79990cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:17.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.358+0000 7f79990cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:17.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.358+0000 7f79990cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:21:17.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.358+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79940627b0 con 0x7f799413f710 2024-03-25T19:21:17.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.687+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f797c0038a0 con 0x7f799413f710 2024-03-25T19:21:17.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.687+0000 7f798bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:17.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.687+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f797c003c70 con 0x7f799413f710 2024-03-25T19:21:17.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.687+0000 7f798bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:17.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.688+0000 7f798bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.688+0000 7f798bfff640 10 monclient: dump: 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:17.688 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:17.689 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.687+0000 7f7991d9b640 4 set_mon_vals no callback set 2024-03-25T19:21:17.689 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.688+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f797c003e90 con 0x7f799413f710 2024-03-25T19:21:17.689 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.689+0000 7f7992d9d640 10 monclient: get_auth_request con 0x7f797406de90 auth_method 0 2024-03-25T19:21:17.689 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.689+0000 7f798bfff640 10 client.4272.objecter ms_handle_connect 0x7f797406de90 2024-03-25T19:21:17.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f797c054940 con 0x7f799413f710 2024-03-25T19:21:17.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 10 client.4272.objecter ms_dispatch 0x7f7994000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:17.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:17.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:17.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-25T19:21:17.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f797c050450 con 0x7f799413f710 2024-03-25T19:21:17.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:17.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.694+0000 7f798bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.931+0000 7f79990cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.931+0000 7f79990cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.931+0000 7f79990cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:21:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.931+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f799413db40 con 0x7f799413f710 2024-03-25T19:21:17.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.985+0000 7f798bfff640 1 -- 172.21.15.60:0/1665592224 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f797c052380 con 0x7f799413f710 2024-03-25T19:21:17.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.985+0000 7f798bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:17.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.985+0000 7f798bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:17.985 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:17.985 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.987+0000 7f79990cf640 10 librados: watch_flush enter 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 10 librados: watch_flush exit 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 10 client.4272.objecter successfully canceled tick 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 >> v1:172.21.15.90:6800/1827638754 conn(0x7f797406de90 legacy=0x7f7974070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 10 monclient: shutdown 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 >> v1:172.21.15.60:6789/0 conn(0x7f799413f710 legacy=0x7f79941df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 shutdown_connections 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 >> 172.21.15.60:0/1665592224 conn(0x7f79940b7ea0 msgr2=0x7f799413d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 shutdown_connections 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 1 -- 172.21.15.60:0/1665592224 wait complete. 2024-03-25T19:21:17.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:17.988+0000 7f79990cf640 1 librados: shutdown 2024-03-25T19:21:18.000 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:24.805 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:24.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.926+0000 7f7a5491c640 10 monclient: get_monmap_and_config 2024-03-25T19:21:24.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.926+0000 7f7a5491c640 10 monclient: build_initial_monmap 2024-03-25T19:21:24.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.926+0000 7f7a5491c640 10 monclient: monmap: 2024-03-25T19:21:24.925 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:24.926974+0000 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:24.926974+0000 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:24.926 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:24.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 1 Processor -- start 2024-03-25T19:21:24.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 1 -- start start 2024-03-25T19:21:24.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 10 monclient: init 2024-03-25T19:21:24.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:24.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 10 monclient(hunting): picked mon.noname-c con 0x7f7a50061830 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 10 monclient(hunting): picked mon.noname-a con 0x7f7a5006ff10 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a50072ae0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.927+0000 7f7a5491c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a50072f90 con 0x7f7a5006ff10 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.928+0000 7f7a5491c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a50070bb0 con 0x7f7a50061830 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.928+0000 7f7a5491c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a500719f0 con 0x7f7a50072ae0 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.928+0000 7f7a5491c640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:24.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.928+0000 7f7a5491c640 10 monclient(hunting): authenticate will time out at 6412.025939s 2024-03-25T19:21:24.940 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.940+0000 7f7a4e59c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7a50061830 0x7f7a500723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:47200/0 (socket says 172.21.15.60:47200) 2024-03-25T19:21:24.940 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.940+0000 7f7a4e59c640 1 -- 172.21.15.60:0/269459777 learned_addr learned my addr 172.21.15.60:0/269459777 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:24.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.953+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124508359 0 0) 0x7f7a50072f90 con 0x7f7a5006ff10 2024-03-25T19:21:24.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.953+0000 7f7a4d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:24.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.953+0000 7f7a4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:24.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.953+0000 7f7a4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:24.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.953+0000 7f7a4d59a640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:21:24.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.953+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a3c003520 con 0x7f7a5006ff10 2024-03-25T19:21:24.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.956+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117984476 0 0) 0x7f7a50070bb0 con 0x7f7a50061830 2024-03-25T19:21:24.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.956+0000 7f7a4d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:24.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.956+0000 7f7a4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:24.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.956+0000 7f7a4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:24.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.956+0000 7f7a4d59a640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:21:24.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.956+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a50072f90 con 0x7f7a50061830 2024-03-25T19:21:24.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2372998337 0 0) 0x7f7a3c003520 con 0x7f7a5006ff10 2024-03-25T19:21:24.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 >> v1:172.21.15.60:6790/0 conn(0x7f7a50072ae0 legacy=0x7f7a5013c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:24.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 >> v1:172.21.15.90:6789/0 conn(0x7f7a50061830 legacy=0x7f7a500723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:24.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:21:24.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 1 monclient: found mon.noname-a 2024-03-25T19:21:24.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:24.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a50071cd0 con 0x7f7a5006ff10 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 10 monclient: _finish_auth 0 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7a38001690 con 0x7f7a5006ff10 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.964+0000 7f7a4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.965+0000 7f7a4d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.965+0000 7f7a4d59a640 10 monclient: dump: 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:24.965 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:24.966 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:24.966 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:24.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.965+0000 7f7a5491c640 5 monclient: authenticate success, global_id 4275 2024-03-25T19:21:24.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.965+0000 7f7a5491c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:24.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.969+0000 7f7a4d59a640 1 -- 172.21.15.60:0/269459777 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a380017f0 con 0x7f7a5006ff10 2024-03-25T19:21:24.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.969+0000 7f7a4d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:24.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.969+0000 7f7a5491c640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:24.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.969+0000 7f7a5491c640 4 set_mon_vals no callback set 2024-03-25T19:21:24.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.970+0000 7f7a5491c640 10 monclient: shutdown 2024-03-25T19:21:24.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.970+0000 7f7a5491c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:24.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.970+0000 7f7a5491c640 1 -- 172.21.15.60:0/269459777 >> v1:172.21.15.60:6789/0 conn(0x7f7a5006ff10 legacy=0x7f7a5013bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:24.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.970+0000 7f7a5491c640 1 -- 172.21.15.60:0/269459777 shutdown_connections 2024-03-25T19:21:24.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.970+0000 7f7a5491c640 1 -- 172.21.15.60:0/269459777 >> 172.21.15.60:0/269459777 conn(0x7f7a500b11e0 msgr2=0x7f7a500b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:24.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.970+0000 7f7a5491c640 1 -- 172.21.15.60:0/269459777 shutdown_connections 2024-03-25T19:21:24.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.970+0000 7f7a5491c640 1 -- 172.21.15.60:0/269459777 wait complete. 2024-03-25T19:21:24.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.971+0000 7f7a5491c640 10 monclient: build_initial_monmap 2024-03-25T19:21:24.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.971+0000 7f7a5491c640 10 monclient: monmap: 2024-03-25T19:21:24.971 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:24.971 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:24.971 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:24.971 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:24.971 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 librados: starting msgr at 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 librados: starting objecter 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 Processor -- start 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 -- start start 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 librados: setting wanted keys 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 librados: calling monclient init 2024-03-25T19:21:24.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient: init 2024-03-25T19:21:24.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:24.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:21:24.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient(hunting): picked mon.noname-a con 0x7f7a50061830 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:24.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5006ff10 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:24.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient(hunting): picked mon.noname-c con 0x7f7a50072ae0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:24.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a50131cc0 con 0x7f7a50061830 2024-03-25T19:21:24.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a50131fa0 con 0x7f7a5006ff10 2024-03-25T19:21:24.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a5013ec20 con 0x7f7a50072ae0 2024-03-25T19:21:24.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:24.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.972+0000 7f7a5491c640 10 monclient(hunting): authenticate will time out at 6412.070817s 2024-03-25T19:21:24.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.980+0000 7f7a4dd9b640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7a5006ff10 0x7f7a5013f140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:47224/0 (socket says 172.21.15.60:47224) 2024-03-25T19:21:24.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.980+0000 7f7a4dd9b640 1 -- 172.21.15.60:0/230803308 learned_addr learned my addr 172.21.15.60:0/230803308 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:24.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.981+0000 7f7a46ffd640 10 client.?.objecter ms_handle_connect 0x7f7a5006ff10 2024-03-25T19:21:24.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.981+0000 7f7a46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:24.991 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.991+0000 7f7a46ffd640 10 client.?.objecter ms_handle_connect 0x7f7a50072ae0 2024-03-25T19:21:24.991 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.991+0000 7f7a46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:24.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.993+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610642562 0 0) 0x7f7a50131fa0 con 0x7f7a5006ff10 2024-03-25T19:21:24.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.993+0000 7f7a46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:24.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.993+0000 7f7a46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:24.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.993+0000 7f7a46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:24.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.993+0000 7f7a46ffd640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:21:24.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.993+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30004eb0 con 0x7f7a5006ff10 2024-03-25T19:21:24.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.999+0000 7f7a46ffd640 10 client.?.objecter ms_handle_connect 0x7f7a50061830 2024-03-25T19:21:24.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:24.999+0000 7f7a46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:25.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.005+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705104595 0 0) 0x7f7a5013ec20 con 0x7f7a50072ae0 2024-03-25T19:21:25.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.005+0000 7f7a46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:25.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.005+0000 7f7a46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:25.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.005+0000 7f7a46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:25.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.005+0000 7f7a46ffd640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:21:25.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.005+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a50131fa0 con 0x7f7a50072ae0 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.016+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3938742651 0 0) 0x7f7a30004eb0 con 0x7f7a5006ff10 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.016+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a5013ec20 con 0x7f7a5006ff10 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.016+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7a38001d20 con 0x7f7a5006ff10 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.016+0000 7f7a46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.016+0000 7f7a46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.016+0000 7f7a46ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:25.016 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:25.017 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:25.017 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:25.017 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:25.017 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:25.017 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:25.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.016+0000 7f7a46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:25.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.019+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774289361 0 0) 0x7f7a50131cc0 con 0x7f7a50061830 2024-03-25T19:21:25.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.019+0000 7f7a46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:25.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.019+0000 7f7a46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:25.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.019+0000 7f7a46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:25.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.019+0000 7f7a46ffd640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:21:25.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.019+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30004eb0 con 0x7f7a50061830 2024-03-25T19:21:25.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1473628891 0 0) 0x7f7a5013ec20 con 0x7f7a5006ff10 2024-03-25T19:21:25.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 >> v1:172.21.15.60:6790/0 conn(0x7f7a50072ae0 legacy=0x7f7a5013f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:25.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 >> v1:172.21.15.60:6789/0 conn(0x7f7a50061830 legacy=0x7f7a50138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:25.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 1 monclient: found mon.b 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a5013de40 con 0x7f7a5006ff10 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 10 monclient: _finish_auth 0 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.031+0000 7f7a46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 10 monclient: _renew_subs 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5013e840 con 0x7f7a5006ff10 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 10 monclient: _renew_subs 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:25.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a501e11c0 con 0x7f7a5006ff10 2024-03-25T19:21:25.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.032+0000 7f7a5491c640 1 librados: init done 2024-03-25T19:21:25.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.033+0000 7f7a5491c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:25.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.033+0000 7f7a5491c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:25.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.033+0000 7f7a5491c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:25.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.033+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5013ea80 con 0x7f7a5006ff10 2024-03-25T19:21:25.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.324+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a380038a0 con 0x7f7a5006ff10 2024-03-25T19:21:25.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.324+0000 7f7a46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:25.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.325+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7a38003ad0 con 0x7f7a5006ff10 2024-03-25T19:21:25.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.325+0000 7f7a46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.325+0000 7f7a46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.325+0000 7f7a46ffd640 10 monclient: dump: 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:25.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.325+0000 7f7a4cd99640 4 set_mon_vals no callback set 2024-03-25T19:21:25.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.325+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f7a38003cf0 con 0x7f7a5006ff10 2024-03-25T19:21:25.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.326+0000 7f7a4e59c640 10 monclient: get_auth_request con 0x7f7a3006de00 auth_method 0 2024-03-25T19:21:25.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.326+0000 7f7a46ffd640 10 client.4264.objecter ms_handle_connect 0x7f7a3006de00 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f7a38054840 con 0x7f7a5006ff10 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 10 client.4264.objecter ms_dispatch 0x7f7a50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a38050350 con 0x7f7a5006ff10 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:25.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.331+0000 7f7a46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:25.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.593+0000 7f7a5491c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:25.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.593+0000 7f7a5491c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:25.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.593+0000 7f7a5491c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:25.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.593+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a500627b0 con 0x7f7a5006ff10 2024-03-25T19:21:25.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.628+0000 7f7a46ffd640 1 -- 172.21.15.60:0/230803308 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f7a38052280 con 0x7f7a5006ff10 2024-03-25T19:21:25.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.628+0000 7f7a46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:25.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.628+0000 7f7a46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:25.628 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:25.628 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 10 librados: watch_flush enter 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 10 librados: watch_flush exit 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 10 client.4264.objecter successfully canceled tick 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 >> v1:172.21.15.90:6800/1827638754 conn(0x7f7a3006de00 legacy=0x7f7a300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 10 monclient: shutdown 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:25.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 >> v1:172.21.15.90:6789/0 conn(0x7f7a5006ff10 legacy=0x7f7a5013f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:25.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 shutdown_connections 2024-03-25T19:21:25.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 >> 172.21.15.60:0/230803308 conn(0x7f7a500b11e0 msgr2=0x7f7a50074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:25.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 shutdown_connections 2024-03-25T19:21:25.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.631+0000 7f7a5491c640 1 -- 172.21.15.60:0/230803308 wait complete. 2024-03-25T19:21:25.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:25.632+0000 7f7a5491c640 1 librados: shutdown 2024-03-25T19:21:25.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:32.447 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:32.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.561+0000 7fb3b5c9a640 10 monclient: get_monmap_and_config 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.561+0000 7fb3b5c9a640 10 monclient: build_initial_monmap 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.561+0000 7fb3b5c9a640 10 monclient: monmap: 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:32.563006+0000 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:32.563006+0000 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:32.562 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.562+0000 7fb3b5c9a640 1 Processor -- start 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.562+0000 7fb3b5c9a640 1 -- start start 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.562+0000 7fb3b5c9a640 10 monclient: init 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b013fc60 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b0070570 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b00727a0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b013c9f0 con 0x7fb3b013fc60 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b006fac0 con 0x7fb3b0070570 2024-03-25T19:21:32.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b00715d0 con 0x7fb3b00727a0 2024-03-25T19:21:32.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:32.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.563+0000 7fb3b5c9a640 10 monclient(hunting): authenticate will time out at 6419.662419s 2024-03-25T19:21:32.571 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.571+0000 7fb3aeffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb3b0070570 0x7fb3b0072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:41478/0 (socket says 172.21.15.60:41478) 2024-03-25T19:21:32.572 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.571+0000 7fb3aeffd640 1 -- 172.21.15.60:0/3719785523 learned_addr learned my addr 172.21.15.60:0/3719785523 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:32.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.644+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904893645 0 0) 0x7fb3b013c9f0 con 0x7fb3b013fc60 2024-03-25T19:21:32.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.644+0000 7fb3ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:21:32.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.644+0000 7fb3ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:32.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.644+0000 7fb3ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:32.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.644+0000 7fb3ae7fc640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:21:32.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.644+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb398003520 con 0x7fb3b013fc60 2024-03-25T19:21:32.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741609807 0 0) 0x7fb398003520 con 0x7fb3b013fc60 2024-03-25T19:21:32.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 >> v1:172.21.15.60:6790/0 conn(0x7fb3b00727a0 legacy=0x7fb3b014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:32.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 >> v1:172.21.15.90:6789/0 conn(0x7fb3b0070570 legacy=0x7fb3b0072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:32.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:21:32.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 1 monclient: found mon.noname-a 2024-03-25T19:21:32.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:32.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b00718b0 con 0x7fb3b013fc60 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb3a0001690 con 0x7fb3b013fc60 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3ae7fc640 10 monclient: dump: 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.654 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:32.655 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:32.655 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:32.655 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:32.655 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:32.655 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:32.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3b5c9a640 5 monclient: authenticate success, global_id 4281 2024-03-25T19:21:32.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.652+0000 7fb3b5c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3a00017f0 con 0x7fb3b013fc60 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3ae7fc640 1 -- 172.21.15.60:0/3719785523 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb3a0001c00 con 0x7fb3b013fc60 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3ae7fc640 10 monclient: dump: 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.662 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3b5c9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:32.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.661+0000 7fb3b5c9a640 4 set_mon_vals no callback set 2024-03-25T19:21:32.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.662+0000 7fb3b5c9a640 10 monclient: shutdown 2024-03-25T19:21:32.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.662+0000 7fb3b5c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:32.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.662+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/3719785523 >> v1:172.21.15.60:6789/0 conn(0x7fb3b013fc60 legacy=0x7fb3b006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:32.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.662+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/3719785523 shutdown_connections 2024-03-25T19:21:32.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.662+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/3719785523 >> 172.21.15.60:0/3719785523 conn(0x7fb3b00b4a80 msgr2=0x7fb3b00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:32.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.662+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/3719785523 shutdown_connections 2024-03-25T19:21:32.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.662+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/3719785523 wait complete. 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.663+0000 7fb3b5c9a640 10 monclient: build_initial_monmap 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.663+0000 7fb3b5c9a640 10 monclient: monmap: 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.663+0000 7fb3b5c9a640 1 librados: starting msgr at 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.663+0000 7fb3b5c9a640 1 librados: starting objecter 2024-03-25T19:21:32.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.663+0000 7fb3b5c9a640 1 Processor -- start 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 1 -- start start 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 1 librados: setting wanted keys 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 1 librados: calling monclient init 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient: init 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b0070570 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b00727a0 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b013fc60 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b0130cf0 con 0x7fb3b013fc60 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b0135920 con 0x7fb3b00727a0 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b0138d30 con 0x7fb3b0070570 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:32.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.664+0000 7fb3b5c9a640 10 monclient(hunting): authenticate will time out at 6419.763620s 2024-03-25T19:21:32.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.676+0000 7fb3affff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fb3b013fc60 0x7fb3b0134900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53854/0 (socket says 172.21.15.60:53854) 2024-03-25T19:21:32.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.676+0000 7fb3affff640 1 -- 172.21.15.60:0/2724631364 learned_addr learned my addr 172.21.15.60:0/2724631364 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:32.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.678+0000 7fb397fff640 10 client.?.objecter ms_handle_connect 0x7fb3b013fc60 2024-03-25T19:21:32.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.678+0000 7fb397fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:32.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.704+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798060918 0 0) 0x7fb3b00715d0 con 0x7fb3b013fc60 2024-03-25T19:21:32.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.704+0000 7fb397fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:32.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.704+0000 7fb397fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:32.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.704+0000 7fb397fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:32.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.704+0000 7fb397fff640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:21:32.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.704+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb39c004eb0 con 0x7fb3b013fc60 2024-03-25T19:21:32.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.706+0000 7fb397fff640 10 client.?.objecter ms_handle_connect 0x7fb3b0070570 2024-03-25T19:21:32.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.706+0000 7fb397fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:32.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.711+0000 7fb397fff640 10 client.?.objecter ms_handle_connect 0x7fb3b00727a0 2024-03-25T19:21:32.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.711+0000 7fb397fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:32.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.718+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352107649 0 0) 0x7fb3b0135920 con 0x7fb3b00727a0 2024-03-25T19:21:32.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.718+0000 7fb397fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:32.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.718+0000 7fb397fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:32.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.718+0000 7fb397fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:32.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.718+0000 7fb397fff640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:21:32.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.718+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b00715d0 con 0x7fb3b00727a0 2024-03-25T19:21:32.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.721+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192190258 0 0) 0x7fb3b0138d30 con 0x7fb3b0070570 2024-03-25T19:21:32.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.721+0000 7fb397fff640 10 monclient(hunting): _negotiate 2024-03-25T19:21:32.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.721+0000 7fb397fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:32.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.721+0000 7fb397fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:32.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.721+0000 7fb397fff640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:21:32.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.721+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b0135920 con 0x7fb3b0070570 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.727+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619201264 0 0) 0x7fb3b00715d0 con 0x7fb3b00727a0 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.727+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b0138d30 con 0x7fb3b00727a0 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.727+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb3a4001800 con 0x7fb3b00727a0 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.727+0000 7fb397fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.727+0000 7fb397fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.727+0000 7fb397fff640 10 monclient(hunting): dump: 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.728 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.727+0000 7fb397fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.728+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1162135638 0 0) 0x7fb3b0135920 con 0x7fb3b0070570 2024-03-25T19:21:32.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.728+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b00715d0 con 0x7fb3b0070570 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.728+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb3a0001bc0 con 0x7fb3b0070570 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.728+0000 7fb397fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.728+0000 7fb397fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.728+0000 7fb397fff640 10 monclient(hunting): dump: 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:32.730 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:32.731 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:32.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.735+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 508557049 0 0) 0x7fb39c004eb0 con 0x7fb3b013fc60 2024-03-25T19:21:32.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.735+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b0135920 con 0x7fb3b013fc60 2024-03-25T19:21:32.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.735+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb3a8001ad0 con 0x7fb3b013fc60 2024-03-25T19:21:32.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.735+0000 7fb397fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:32.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.735+0000 7fb397fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.735+0000 7fb397fff640 10 monclient(hunting): dump: 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:32.736 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3682693926 0 0) 0x7fb3b00715d0 con 0x7fb3b0070570 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 >> v1:172.21.15.90:6789/0 conn(0x7fb3b00727a0 legacy=0x7fb3b01341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 >> v1:172.21.15.60:6789/0 conn(0x7fb3b013fc60 legacy=0x7fb3b0134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 10 monclient: _finish_hunting 0 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 1 monclient: found mon.c 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0139010 con 0x7fb3b0070570 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 10 monclient: _finish_auth 0 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 10 monclient: _check_auth_tickets 2024-03-25T19:21:32.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb397fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 5 monclient: authenticate success, global_id 4268 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 10 monclient: _renew_subs 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b0139e90 con 0x7fb3b0070570 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 10 monclient: _renew_subs 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b01379b0 con 0x7fb3b0070570 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.739+0000 7fb3b5c9a640 1 librados: init done 2024-03-25T19:21:32.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.740+0000 7fb3b5c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:32.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.740+0000 7fb3b5c9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:32.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.740+0000 7fb3b5c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:21:32.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:32.740+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b00627b0 con 0x7fb3b0070570 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.065+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3a00038a0 con 0x7fb3b0070570 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.065+0000 7fb397fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.065+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb3a0003bb0 con 0x7fb3b0070570 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.065+0000 7fb397fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.065+0000 7fb397fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.065+0000 7fb397fff640 10 monclient: dump: 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:33.066 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:33.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.065+0000 7fb3adffb640 4 set_mon_vals no callback set 2024-03-25T19:21:33.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.066+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fb3a0019b80 con 0x7fb3b0070570 2024-03-25T19:21:33.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.067+0000 7fb3aeffd640 10 monclient: get_auth_request con 0x7fb39c06e160 auth_method 0 2024-03-25T19:21:33.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.067+0000 7fb397fff640 10 client.4268.objecter ms_handle_connect 0x7fb39c06e160 2024-03-25T19:21:33.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fb3a00536b0 con 0x7fb3b0070570 2024-03-25T19:21:33.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 10 client.4268.objecter ms_dispatch 0x7fb3b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:33.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:33.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:33.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-25T19:21:33.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3a004f1c0 con 0x7fb3b0070570 2024-03-25T19:21:33.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:33.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.073+0000 7fb397fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:33.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.337+0000 7fb3b5c9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:33.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.337+0000 7fb3b5c9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:33.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.337+0000 7fb3b5c9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:21:33.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.337+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3b00af160 con 0x7fb3b0070570 2024-03-25T19:21:33.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.367+0000 7fb397fff640 1 -- 172.21.15.60:0/2724631364 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fb3a00510f0 con 0x7fb3b0070570 2024-03-25T19:21:33.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.368+0000 7fb397fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:33.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.368+0000 7fb397fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:33.368 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:33.368 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:33.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 10 librados: watch_flush enter 2024-03-25T19:21:33.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 10 librados: watch_flush exit 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 10 client.4268.objecter successfully canceled tick 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 >> v1:172.21.15.90:6800/1827638754 conn(0x7fb39c06e160 legacy=0x7fb39c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 10 monclient: shutdown 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 >> v1:172.21.15.60:6790/0 conn(0x7fb3b0070570 legacy=0x7fb3b0139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 shutdown_connections 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.370+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 >> 172.21.15.60:0/2724631364 conn(0x7fb3b00b4a80 msgr2=0x7fb3b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.371+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 shutdown_connections 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.371+0000 7fb3b5c9a640 1 -- 172.21.15.60:0/2724631364 wait complete. 2024-03-25T19:21:33.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:33.371+0000 7fb3b5c9a640 1 librados: shutdown 2024-03-25T19:21:33.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:33.840 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:34.552 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.847 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:34.558 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.889 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:34.600 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.892 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:34.603 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.896 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:34.607 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.912 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:34.566 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.923 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:34.577 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.940 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:34.652 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:33.980 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:34.634 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:40.189 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:40.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.305+0000 7f8b68a74640 10 monclient: get_monmap_and_config 2024-03-25T19:21:40.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.305+0000 7f8b68a74640 10 monclient: build_initial_monmap 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.305+0000 7f8b68a74640 10 monclient: monmap: 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:40.307006+0000 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:40.307006+0000 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:40.306 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.305+0000 7f8b68a74640 1 Processor -- start 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.305+0000 7f8b68a74640 1 -- start start 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.305+0000 7f8b68a74640 10 monclient: init 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.306+0000 7f8b68a74640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.306+0000 7f8b68a74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.306+0000 7f8b68a74640 10 monclient(hunting): picked mon.noname-b con 0x7f8b640a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.306+0000 7f8b68a74640 10 monclient(hunting): picked mon.noname-c con 0x7f8b6413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.306+0000 7f8b68a74640 10 monclient(hunting): picked mon.noname-a con 0x7f8b641430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.306+0000 7f8b68a74640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64142520 con 0x7f8b641430c0 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.307+0000 7f8b68a74640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640b28a0 con 0x7f8b6413f710 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.307+0000 7f8b68a74640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640b2a30 con 0x7f8b640a98f0 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.307+0000 7f8b68a74640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:40.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.307+0000 7f8b68a74640 10 monclient(hunting): authenticate will time out at 6427.405983s 2024-03-25T19:21:40.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.309+0000 7f8b6259c640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f8b640a98f0 0x7f8b640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:58934/0 (socket says 172.21.15.60:58934) 2024-03-25T19:21:40.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.309+0000 7f8b6259c640 1 -- 172.21.15.60:0/2512104151 learned_addr learned my addr 172.21.15.60:0/2512104151 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:40.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.326+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136730872 0 0) 0x7f8b640b2a30 con 0x7f8b640a98f0 2024-03-25T19:21:40.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.326+0000 7f8b6159a640 10 monclient(hunting): _negotiate 2024-03-25T19:21:40.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.326+0000 7f8b6159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:40.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.326+0000 7f8b6159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:40.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.326+0000 7f8b6159a640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:21:40.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.326+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b4c003520 con 0x7f8b640a98f0 2024-03-25T19:21:40.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3636985430 0 0) 0x7f8b4c003520 con 0x7f8b640a98f0 2024-03-25T19:21:40.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 >> v1:172.21.15.90:6789/0 conn(0x7f8b6413f710 legacy=0x7f8b641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:40.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 >> v1:172.21.15.60:6789/0 conn(0x7f8b641430c0 legacy=0x7f8b6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:40.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 10 monclient: _finish_hunting 0 2024-03-25T19:21:40.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 1 monclient: found mon.noname-b 2024-03-25T19:21:40.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640b44c0 con 0x7f8b640a98f0 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 10 monclient: _finish_auth 0 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 10 monclient: _check_auth_tickets 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8b54001690 con 0x7f8b640a98f0 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b6159a640 10 monclient: dump: 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.345 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:40.346 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:40.346 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:40.346 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:40.346 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:40.346 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:40.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b68a74640 5 monclient: authenticate success, global_id 4271 2024-03-25T19:21:40.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.343+0000 7f8b68a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:40.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.351+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b540017f0 con 0x7f8b640a98f0 2024-03-25T19:21:40.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.351+0000 7f8b6159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:40.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.351+0000 7f8b6159a640 1 -- 172.21.15.60:0/2512104151 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8b54001c00 con 0x7f8b640a98f0 2024-03-25T19:21:40.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.351+0000 7f8b6159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:40.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.351+0000 7f8b6159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:40.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.351+0000 7f8b6159a640 10 monclient: dump: 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.352+0000 7f8b68a74640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:40.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.352+0000 7f8b68a74640 4 set_mon_vals no callback set 2024-03-25T19:21:40.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.352+0000 7f8b68a74640 10 monclient: shutdown 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.352+0000 7f8b68a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.352+0000 7f8b68a74640 1 -- 172.21.15.60:0/2512104151 >> v1:172.21.15.60:6790/0 conn(0x7f8b640a98f0 legacy=0x7f8b640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.353+0000 7f8b68a74640 1 -- 172.21.15.60:0/2512104151 shutdown_connections 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.353+0000 7f8b68a74640 1 -- 172.21.15.60:0/2512104151 >> 172.21.15.60:0/2512104151 conn(0x7f8b640b7ea0 msgr2=0x7f8b640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.353+0000 7f8b68a74640 1 -- 172.21.15.60:0/2512104151 shutdown_connections 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.353+0000 7f8b68a74640 1 -- 172.21.15.60:0/2512104151 wait complete. 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 10 monclient: build_initial_monmap 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 10 monclient: monmap: 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:40.355 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 1 librados: starting msgr at 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 1 librados: starting objecter 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 1 Processor -- start 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 1 -- start start 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 1 librados: setting wanted keys 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 1 librados: calling monclient init 2024-03-25T19:21:40.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.354+0000 7f8b68a74640 10 monclient: init 2024-03-25T19:21:40.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:40.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:21:40.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 10 monclient(hunting): picked mon.noname-a con 0x7f8b640a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:40.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 10 monclient(hunting): picked mon.noname-b con 0x7f8b6413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:40.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 10 monclient(hunting): picked mon.noname-c con 0x7f8b641430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:40.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64139a20 con 0x7f8b640a98f0 2024-03-25T19:21:40.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64138ba0 con 0x7f8b6413f710 2024-03-25T19:21:40.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b64135240 con 0x7f8b641430c0 2024-03-25T19:21:40.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:40.358 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.355+0000 7f8b68a74640 10 monclient(hunting): authenticate will time out at 6427.454247s 2024-03-25T19:21:40.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.368+0000 7f8b62d9d640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f8b641430c0 0x7f8b641e2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:58972/0 (socket says 172.21.15.60:58972) 2024-03-25T19:21:40.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.368+0000 7f8b62d9d640 1 -- 172.21.15.60:0/2297282596 learned_addr learned my addr 172.21.15.60:0/2297282596 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:40.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.369+0000 7f8b52ffd640 10 client.?.objecter ms_handle_connect 0x7f8b6413f710 2024-03-25T19:21:40.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.369+0000 7f8b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:40.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.370+0000 7f8b52ffd640 10 client.?.objecter ms_handle_connect 0x7f8b641430c0 2024-03-25T19:21:40.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.370+0000 7f8b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:40.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.376+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095293840 0 0) 0x7f8b64138ba0 con 0x7f8b6413f710 2024-03-25T19:21:40.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.377+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:40.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.377+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:40.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.377+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:40.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.377+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:21:40.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.377+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b44004eb0 con 0x7f8b6413f710 2024-03-25T19:21:40.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.381+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298677005 0 0) 0x7f8b64135240 con 0x7f8b641430c0 2024-03-25T19:21:40.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.381+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:40.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.381+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:40.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.381+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:40.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.381+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:21:40.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.381+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b64138ba0 con 0x7f8b641430c0 2024-03-25T19:21:40.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.383+0000 7f8b52ffd640 10 client.?.objecter ms_handle_connect 0x7f8b640a98f0 2024-03-25T19:21:40.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.383+0000 7f8b52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:40.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.384+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 810139356 0 0) 0x7f8b44004eb0 con 0x7f8b6413f710 2024-03-25T19:21:40.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.384+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b64135240 con 0x7f8b6413f710 2024-03-25T19:21:40.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.384+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8b58001930 con 0x7f8b6413f710 2024-03-25T19:21:40.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.384+0000 7f8b52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:40.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.385+0000 7f8b52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:40.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.385+0000 7f8b52ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:40.386 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.385+0000 7f8b52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:40.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.395+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982076432 0 0) 0x7f8b64139a20 con 0x7f8b640a98f0 2024-03-25T19:21:40.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.395+0000 7f8b52ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:21:40.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.395+0000 7f8b52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:40.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.395+0000 7f8b52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:40.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.395+0000 7f8b52ffd640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:21:40.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.395+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b44004eb0 con 0x7f8b640a98f0 2024-03-25T19:21:40.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.402+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 425689769 0 0) 0x7f8b64138ba0 con 0x7f8b641430c0 2024-03-25T19:21:40.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.402+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b64139a20 con 0x7f8b641430c0 2024-03-25T19:21:40.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.402+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8b5c001bf0 con 0x7f8b641430c0 2024-03-25T19:21:40.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.402+0000 7f8b52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:40.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.402+0000 7f8b52ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.402+0000 7f8b52ffd640 10 monclient(hunting): dump: 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:40.403 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3180781600 0 0) 0x7f8b64135240 con 0x7f8b6413f710 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 >> v1:172.21.15.60:6790/0 conn(0x7f8b641430c0 legacy=0x7f8b641e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 >> v1:172.21.15.60:6789/0 conn(0x7f8b640a98f0 legacy=0x7f8b64138490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 1 monclient: found mon.b 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:40.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b641353d0 con 0x7f8b6413f710 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 10 monclient: _finish_auth 0 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 5 monclient: authenticate success, global_id 4276 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 10 monclient: _renew_subs 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b641d5020 con 0x7f8b6413f710 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 10 monclient: _renew_subs 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b64137eb0 con 0x7f8b6413f710 2024-03-25T19:21:40.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.413+0000 7f8b68a74640 1 librados: init done 2024-03-25T19:21:40.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.414+0000 7f8b68a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:40.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.414+0000 7f8b68a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:40.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.414+0000 7f8b68a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:40.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.414+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b641d41e0 con 0x7f8b6413f710 2024-03-25T19:21:40.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8b580035f0 con 0x7f8b6413f710 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8b58003980 con 0x7f8b6413f710 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b52ffd640 10 monclient: dump: 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:40.734 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:40.735 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:40.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b60d99640 4 set_mon_vals no callback set 2024-03-25T19:21:40.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.733+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f8b58003ba0 con 0x7f8b6413f710 2024-03-25T19:21:40.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.734+0000 7f8b6259c640 10 monclient: get_auth_request con 0x7f8b4406dfb0 auth_method 0 2024-03-25T19:21:40.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.734+0000 7f8b52ffd640 10 client.4276.objecter ms_handle_connect 0x7f8b4406dfb0 2024-03-25T19:21:40.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f8b580535b0 con 0x7f8b6413f710 2024-03-25T19:21:40.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 10 client.4276.objecter ms_dispatch 0x7f8b64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:40.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:40.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:40.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T19:21:40.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8b5804f0c0 con 0x7f8b6413f710 2024-03-25T19:21:40.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:40.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.738+0000 7f8b52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:40.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.953+0000 7f8b68a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.953+0000 7f8b68a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.953+0000 7f8b68a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.953+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b640627b0 con 0x7f8b6413f710 2024-03-25T19:21:40.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.979+0000 7f8b52ffd640 1 -- 172.21.15.60:0/2297282596 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f8b58050ff0 con 0x7f8b6413f710 2024-03-25T19:21:40.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.979+0000 7f8b52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:40.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.979+0000 7f8b52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:40.980 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:40.980 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 10 librados: watch_flush enter 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 10 librados: watch_flush exit 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 10 client.4276.objecter successfully canceled tick 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 >> v1:172.21.15.90:6800/1827638754 conn(0x7f8b4406dfb0 legacy=0x7f8b44070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 10 monclient: shutdown 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 >> v1:172.21.15.90:6789/0 conn(0x7f8b6413f710 legacy=0x7f8b641d4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:40.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 shutdown_connections 2024-03-25T19:21:40.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 >> 172.21.15.60:0/2297282596 conn(0x7f8b640b7ea0 msgr2=0x7f8b64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:40.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 shutdown_connections 2024-03-25T19:21:40.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 1 -- 172.21.15.60:0/2297282596 wait complete. 2024-03-25T19:21:40.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:40.982+0000 7f8b68a74640 1 librados: shutdown 2024-03-25T19:21:40.995 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:47.802 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.917+0000 7f8fa2ca8640 10 monclient: get_monmap_and_config 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.917+0000 7f8fa2ca8640 10 monclient: build_initial_monmap 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.917+0000 7f8fa2ca8640 10 monclient: monmap: 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:47.919547+0000 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:47.919547+0000 2024-03-25T19:21:47.918 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:47.919 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:47.919 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:47.919 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:47.919 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:47.919 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:47.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.918+0000 7f8fa2ca8640 1 Processor -- start 2024-03-25T19:21:47.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.918+0000 7f8fa2ca8640 1 -- start start 2024-03-25T19:21:47.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.918+0000 7f8fa2ca8640 10 monclient: init 2024-03-25T19:21:47.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:47.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:47.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f8f9c0a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:47.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f9c13f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:47.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9c1430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:47.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c142520 con 0x7f8f9c13f710 2024-03-25T19:21:47.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0b28a0 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0b2a30 con 0x7f8f9c1430c0 2024-03-25T19:21:47.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:47.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.919+0000 7f8fa2ca8640 10 monclient(hunting): authenticate will time out at 6435.018572s 2024-03-25T19:21:47.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.924+0000 7f8fa0a44640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8f9c0a98f0 0x7f8f9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:58816/0 (socket says 172.21.15.60:58816) 2024-03-25T19:21:47.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.924+0000 7f8fa0a44640 1 -- 172.21.15.60:0/1856753146 learned_addr learned my addr 172.21.15.60:0/1856753146 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:47.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.935+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174746328 0 0) 0x7f8f9c0b28a0 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.936+0000 7f8f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:47.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.936+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:47.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.936+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:47.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.936+0000 7f8f9b7fe640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:21:47.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.936+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c003520 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.945+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755678998 0 0) 0x7f8f9c0b2a30 con 0x7f8f9c1430c0 2024-03-25T19:21:47.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.945+0000 7f8f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:47.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.945+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:47.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.945+0000 7f8f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:47.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.945+0000 7f8f9b7fe640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:21:47.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.945+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c0b28a0 con 0x7f8f9c1430c0 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 471995113 0 0) 0x7f8f8c003520 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 >> v1:172.21.15.60:6790/0 conn(0x7f8f9c1430c0 legacy=0x7f8f9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 >> v1:172.21.15.60:6789/0 conn(0x7f8f9c13f710 legacy=0x7f8f9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c0b44c0 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8f90001690 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8f9b7fe640 10 monclient: dump: 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8fa2ca8640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:21:47.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.957+0000 7f8fa2ca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:47.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f900017f0 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8f9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8f9b7fe640 1 -- 172.21.15.60:0/1856753146 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8f90001c00 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8fa2ca8640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8fa2ca8640 4 set_mon_vals no callback set 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8f9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.972+0000 7f8f9b7fe640 10 monclient: dump: 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:47.973 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:47.974 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:47.974 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:47.974 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:47.974 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:47.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.973+0000 7f8fa2ca8640 10 monclient: shutdown 2024-03-25T19:21:47.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.973+0000 7f8fa2ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.973+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/1856753146 >> v1:172.21.15.90:6789/0 conn(0x7f8f9c0a98f0 legacy=0x7f8f9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.973+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/1856753146 shutdown_connections 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.973+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/1856753146 >> 172.21.15.60:0/1856753146 conn(0x7f8f9c0b7ea0 msgr2=0x7f8f9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.973+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/1856753146 shutdown_connections 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.973+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/1856753146 wait complete. 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.974+0000 7f8fa2ca8640 10 monclient: build_initial_monmap 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.974+0000 7f8fa2ca8640 10 monclient: monmap: 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:47.976 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.975+0000 7f8fa2ca8640 1 librados: starting msgr at 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.975+0000 7f8fa2ca8640 1 librados: starting objecter 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.975+0000 7f8fa2ca8640 1 Processor -- start 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.975+0000 7f8fa2ca8640 1 -- start start 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.975+0000 7f8fa2ca8640 1 librados: setting wanted keys 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.975+0000 7f8fa2ca8640 1 librados: calling monclient init 2024-03-25T19:21:47.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.975+0000 7f8fa2ca8640 10 monclient: init 2024-03-25T19:21:47.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:47.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:21:47.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9c0a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:47.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f8f9c13f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:47.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f9c1430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:47.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c135590 con 0x7f8f9c1430c0 2024-03-25T19:21:47.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c136e50 con 0x7f8f9c0a98f0 2024-03-25T19:21:47.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c135070 con 0x7f8f9c13f710 2024-03-25T19:21:47.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:47.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.976+0000 7f8fa2ca8640 10 monclient(hunting): authenticate will time out at 6435.075092s 2024-03-25T19:21:47.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.988+0000 7f8fa1245640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8f9c1430c0 0x7f8f9c1e3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:46976/0 (socket says 172.21.15.60:46976) 2024-03-25T19:21:47.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.988+0000 7f8fa1245640 1 -- 172.21.15.60:0/3406638436 learned_addr learned my addr 172.21.15.60:0/3406638436 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:47.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.988+0000 7f8f997fa640 10 client.?.objecter ms_handle_connect 0x7f8f9c1430c0 2024-03-25T19:21:47.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.988+0000 7f8f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:47.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.993+0000 7f8f997fa640 10 client.?.objecter ms_handle_connect 0x7f8f9c0a98f0 2024-03-25T19:21:47.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.993+0000 7f8f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:47.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.997+0000 7f8f997fa640 10 client.?.objecter ms_handle_connect 0x7f8f9c13f710 2024-03-25T19:21:47.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:47.997+0000 7f8f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:48.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.005+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240949958 0 0) 0x7f8f9c136e50 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.005+0000 7f8f997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:48.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.005+0000 7f8f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:48.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.005+0000 7f8f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:48.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.005+0000 7f8f997fa640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:21:48.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.005+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f80004eb0 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.011+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470039011 0 0) 0x7f8f9c135590 con 0x7f8f9c1430c0 2024-03-25T19:21:48.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.011+0000 7f8f997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:48.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.011+0000 7f8f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:48.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.011+0000 7f8f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:48.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.011+0000 7f8f997fa640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:21:48.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.011+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c136e50 con 0x7f8f9c1430c0 2024-03-25T19:21:48.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.030+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358768112 0 0) 0x7f8f9c135070 con 0x7f8f9c13f710 2024-03-25T19:21:48.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.030+0000 7f8f997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:48.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.030+0000 7f8f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:48.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.030+0000 7f8f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:48.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.030+0000 7f8f997fa640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:21:48.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.030+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c135590 con 0x7f8f9c13f710 2024-03-25T19:21:48.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.033+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 31803448 0 0) 0x7f8f80004eb0 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.033+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f9c135070 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.033+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8f90001b00 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.033+0000 7f8f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.033+0000 7f8f997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.033+0000 7f8f997fa640 10 monclient(hunting): dump: 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:48.034 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:48.035 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.033+0000 7f8f997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.038+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3349461632 0 0) 0x7f8f9c136e50 con 0x7f8f9c1430c0 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.038+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f80004eb0 con 0x7f8f9c1430c0 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.038+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8f94001c50 con 0x7f8f9c1430c0 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.038+0000 7f8f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.038+0000 7f8f997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.038+0000 7f8f997fa640 10 monclient(hunting): dump: 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.039 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.040 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:48.040 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:48.040 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:48.040 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:48.040 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:48.040 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:48.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.052+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3661549840 0 0) 0x7f8f9c135590 con 0x7f8f9c13f710 2024-03-25T19:21:48.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f9c136e50 con 0x7f8f9c13f710 2024-03-25T19:21:48.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8f88001870 con 0x7f8f9c13f710 2024-03-25T19:21:48.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:48.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:21:48.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 10 monclient(hunting): dump: 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1284167118 0 0) 0x7f8f9c135070 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 >> v1:172.21.15.60:6790/0 conn(0x7f8f9c13f710 legacy=0x7f8f9c1d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 >> v1:172.21.15.60:6789/0 conn(0x7f8f9c1430c0 legacy=0x7f8f9c1e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:48.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 10 monclient: _finish_hunting 0 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 1 monclient: found mon.b 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c135d80 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 10 monclient: _finish_auth 0 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 10 monclient: _check_auth_tickets 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8f997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 10 monclient: _renew_subs 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c131660 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 10 monclient: _renew_subs 2024-03-25T19:21:48.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:48.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c0b7a80 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.053+0000 7f8fa2ca8640 1 librados: init done 2024-03-25T19:21:48.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.054+0000 7f8fa2ca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:48.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.054+0000 7f8fa2ca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:48.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.054+0000 7f8fa2ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:48.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.054+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c131a00 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.345+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f90001220 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.345+0000 7f8f997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.346+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8f90004a90 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.345+0000 7f8f9affd640 4 set_mon_vals no callback set 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.346+0000 7f8f997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.346+0000 7f8f997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.346+0000 7f8f997fa640 10 monclient: dump: 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:48.347 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.346+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f8f9001aa60 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.347+0000 7f8f9bfff640 10 monclient: get_auth_request con 0x7f8f8006e0b0 auth_method 0 2024-03-25T19:21:48.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.347+0000 7f8f997fa640 10 client.4282.objecter ms_handle_connect 0x7f8f8006e0b0 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f8f900546e0 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 10 client.4282.objecter ms_dispatch 0x7f8f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f90050020 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:48.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.351+0000 7f8f997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:48.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.604+0000 7f8fa2ca8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:48.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.604+0000 7f8fa2ca8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:48.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.604+0000 7f8fa2ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:48.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.604+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f9c135200 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.651+0000 7f8f997fa640 1 -- 172.21.15.60:0/3406638436 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f8f90052120 con 0x7f8f9c0a98f0 2024-03-25T19:21:48.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.651+0000 7f8f997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.651+0000 7f8f997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:48.652 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:48.652 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 10 librados: watch_flush enter 2024-03-25T19:21:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 10 librados: watch_flush exit 2024-03-25T19:21:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:21:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:21:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 >> v1:172.21.15.90:6800/1827638754 conn(0x7f8f8006e0b0 legacy=0x7f8f80070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 10 monclient: shutdown 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 >> v1:172.21.15.90:6789/0 conn(0x7f8f9c0a98f0 legacy=0x7f8f9c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 shutdown_connections 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 >> 172.21.15.60:0/3406638436 conn(0x7f8f9c0b7ea0 msgr2=0x7f8f9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 shutdown_connections 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 1 -- 172.21.15.60:0/3406638436 wait complete. 2024-03-25T19:21:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:48.653+0000 7f8fa2ca8640 1 librados: shutdown 2024-03-25T19:21:48.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:21:53.876 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:54.587 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:53.930 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:54.641 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:53.967 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:01:54.679 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:53.991 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:54.703 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:53.995 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:54.649 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:54.003 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:54.715 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:54.010 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:54.664 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:54.040 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:01:54.694 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:54.042 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:01:54.753 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:21:55.472 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:21:55.589 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.589+0000 7f43970c9640 10 monclient: get_monmap_and_config 2024-03-25T19:21:55.589 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.589+0000 7f43970c9640 10 monclient: build_initial_monmap 2024-03-25T19:21:55.589 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.589+0000 7f43970c9640 10 monclient: monmap: 2024-03-25T19:21:55.589 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:55.589 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:55.589 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:21:55.590887+0000 2024-03-25T19:21:55.590 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:21:55.590887+0000 2024-03-25T19:21:55.590 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:55.590 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:55.590 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:55.590 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:21:55.590 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:21:55.590 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:55.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 1 Processor -- start 2024-03-25T19:21:55.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 1 -- start start 2024-03-25T19:21:55.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient: init 2024-03-25T19:21:55.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:55.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:55.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient(hunting): picked mon.noname-c con 0x7f43900a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:55.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient(hunting): picked mon.noname-a con 0x7f439013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:55.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient(hunting): picked mon.noname-b con 0x7f43901430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:55.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4390142520 con 0x7f439013f710 2024-03-25T19:21:55.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43900b28a0 con 0x7f43900a98f0 2024-03-25T19:21:55.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43900b2a30 con 0x7f43901430c0 2024-03-25T19:21:55.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:55.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.590+0000 7f43970c9640 10 monclient(hunting): authenticate will time out at 6442.689798s 2024-03-25T19:21:55.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.603+0000 7f4387fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f439013f710 0x7f43901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51606/0 (socket says 172.21.15.60:51606) 2024-03-25T19:21:55.604 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.603+0000 7f4387fff640 1 -- 172.21.15.60:0/3639992077 learned_addr learned my addr 172.21.15.60:0/3639992077 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:55.616 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22965108 0 0) 0x7f43900b28a0 con 0x7f43900a98f0 2024-03-25T19:21:55.616 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:55.616 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:55.616 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4378003520 con 0x7f43900a98f0 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606044419 0 0) 0x7f4390142520 con 0x7f439013f710 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:21:55.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.616+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43900b28a0 con 0x7f439013f710 2024-03-25T19:21:55.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 622340959 0 0) 0x7f4378003520 con 0x7f43900a98f0 2024-03-25T19:21:55.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 >> v1:172.21.15.60:6790/0 conn(0x7f43901430c0 legacy=0x7f439014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:55.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 >> v1:172.21.15.60:6789/0 conn(0x7f439013f710 legacy=0x7f43901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:55.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:21:55.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 1 monclient: found mon.noname-c 2024-03-25T19:21:55.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:55.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43900b44c0 con 0x7f43900a98f0 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 10 monclient: _finish_auth 0 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f438c001690 con 0x7f43900a98f0 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43877fe640 10 monclient: dump: 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:55.628 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:55.629 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:55.629 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43970c9640 5 monclient: authenticate success, global_id 4285 2024-03-25T19:21:55.629 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.626+0000 7f43970c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f438c0017f0 con 0x7f43900a98f0 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43877fe640 1 -- 172.21.15.60:0/3639992077 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f438c001c00 con 0x7f43900a98f0 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43970c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43970c9640 4 set_mon_vals no callback set 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.640+0000 7f43877fe640 10 monclient: dump: 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:55.641 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:55.642 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:55.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.641+0000 7f43970c9640 10 monclient: shutdown 2024-03-25T19:21:55.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.641+0000 7f43970c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:55.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.641+0000 7f43970c9640 1 -- 172.21.15.60:0/3639992077 >> v1:172.21.15.90:6789/0 conn(0x7f43900a98f0 legacy=0x7f43900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:55.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.641+0000 7f43970c9640 1 -- 172.21.15.60:0/3639992077 shutdown_connections 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.641+0000 7f43970c9640 1 -- 172.21.15.60:0/3639992077 >> 172.21.15.60:0/3639992077 conn(0x7f43900b7ea0 msgr2=0x7f43900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.641+0000 7f43970c9640 1 -- 172.21.15.60:0/3639992077 shutdown_connections 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.641+0000 7f43970c9640 1 -- 172.21.15.60:0/3639992077 wait complete. 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.642+0000 7f43970c9640 10 monclient: build_initial_monmap 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.642+0000 7f43970c9640 10 monclient: monmap: 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:21:55.644 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:55.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.642+0000 7f43970c9640 1 librados: starting msgr at 2024-03-25T19:21:55.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.642+0000 7f43970c9640 1 librados: starting objecter 2024-03-25T19:21:55.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.643+0000 7f43970c9640 1 Processor -- start 2024-03-25T19:21:55.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.643+0000 7f43970c9640 1 -- start start 2024-03-25T19:21:55.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.643+0000 7f43970c9640 1 librados: setting wanted keys 2024-03-25T19:21:55.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.643+0000 7f43970c9640 1 librados: calling monclient init 2024-03-25T19:21:55.645 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.643+0000 7f43970c9640 10 monclient: init 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 10 monclient(hunting): picked mon.noname-b con 0x7f43900a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 10 monclient(hunting): picked mon.noname-a con 0x7f439013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 10 monclient(hunting): picked mon.noname-c con 0x7f43901430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43901d7380 con 0x7f439013f710 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4390139840 con 0x7f43900a98f0 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43901d8200 con 0x7f43901430c0 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:21:55.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.644+0000 7f43970c9640 10 monclient(hunting): authenticate will time out at 6442.743118s 2024-03-25T19:21:55.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.661+0000 7f4394e65640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f43900a98f0 0x7f43901d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:35312/0 (socket says 172.21.15.60:35312) 2024-03-25T19:21:55.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.661+0000 7f4394e65640 1 -- 172.21.15.60:0/1326592037 learned_addr learned my addr 172.21.15.60:0/1326592037 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:21:55.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.663+0000 7f43857fa640 10 client.?.objecter ms_handle_connect 0x7f43900a98f0 2024-03-25T19:21:55.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.663+0000 7f43857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:55.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.669+0000 7f43857fa640 10 client.?.objecter ms_handle_connect 0x7f439013f710 2024-03-25T19:21:55.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.669+0000 7f43857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:55.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.673+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791301018 0 0) 0x7f4390139840 con 0x7f43900a98f0 2024-03-25T19:21:55.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.673+0000 7f43857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:55.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.673+0000 7f43857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:55.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.673+0000 7f43857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:55.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.673+0000 7f43857fa640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:21:55.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.673+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437c004eb0 con 0x7f43900a98f0 2024-03-25T19:21:55.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.680+0000 7f43857fa640 10 client.?.objecter ms_handle_connect 0x7f43901430c0 2024-03-25T19:21:55.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.680+0000 7f43857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:21:55.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.683+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781849535 0 0) 0x7f43901d7380 con 0x7f439013f710 2024-03-25T19:21:55.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.683+0000 7f43857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:55.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.683+0000 7f43857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:55.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.683+0000 7f43857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:55.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.683+0000 7f43857fa640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:21:55.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.683+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4390139840 con 0x7f439013f710 2024-03-25T19:21:55.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.687+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883864265 0 0) 0x7f43901d8200 con 0x7f43901430c0 2024-03-25T19:21:55.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.687+0000 7f43857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:21:55.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.687+0000 7f43857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:21:55.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.687+0000 7f43857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:21:55.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.687+0000 7f43857fa640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:21:55.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.687+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43901d7380 con 0x7f43901430c0 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.691+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47901215 0 0) 0x7f437c004eb0 con 0x7f43900a98f0 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.691+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43901d8200 con 0x7f43900a98f0 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.691+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f438c001690 con 0x7f43900a98f0 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.691+0000 7f43857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.691+0000 7f43857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.691+0000 7f43857fa640 10 monclient(hunting): dump: 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:55.692 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:55.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.691+0000 7f43857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:21:55.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 891612248 0 0) 0x7f43901d8200 con 0x7f43900a98f0 2024-03-25T19:21:55.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 >> v1:172.21.15.60:6790/0 conn(0x7f43901430c0 legacy=0x7f43901daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:55.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 >> v1:172.21.15.60:6789/0 conn(0x7f439013f710 legacy=0x7f43901da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:55.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 10 monclient: _finish_hunting 0 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 1 monclient: found mon.b 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43901d75d0 con 0x7f43900a98f0 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 10 monclient: _finish_auth 0 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 10 monclient: _check_auth_tickets 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43970c9640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43970c9640 10 monclient: _renew_subs 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43970c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.703+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43901db6a0 con 0x7f43900a98f0 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.704+0000 7f43970c9640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.704+0000 7f43970c9640 10 monclient: _renew_subs 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.704+0000 7f43970c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:55.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.704+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4390137910 con 0x7f43900a98f0 2024-03-25T19:21:55.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.704+0000 7f43970c9640 1 librados: init done 2024-03-25T19:21:55.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.705+0000 7f43970c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:55.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.705+0000 7f43970c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:55.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.705+0000 7f43970c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:55.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.705+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43900627b0 con 0x7f43900a98f0 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f438c0038a0 con 0x7f43900a98f0 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f43857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f438c003c70 con 0x7f43900a98f0 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f43857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f43857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f43857fa640 10 monclient: dump: 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:21:55.994 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f4386ffd640 4 set_mon_vals no callback set 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.993+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f438c003e90 con 0x7f43900a98f0 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.994+0000 7f4387fff640 10 monclient: get_auth_request con 0x7f437c06ddf0 auth_method 0 2024-03-25T19:21:55.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.994+0000 7f43857fa640 10 client.4288.objecter ms_handle_connect 0x7f437c06ddf0 2024-03-25T19:21:55.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f438c054960 con 0x7f43900a98f0 2024-03-25T19:21:55.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 10 client.4288.objecter ms_dispatch 0x7f4390000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:21:55.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:21:56.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:21:56.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:21:56.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f438c050470 con 0x7f43900a98f0 2024-03-25T19:21:56.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:21:56.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:55.999+0000 7f43857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:21:56.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.256+0000 7f43970c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:56.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.256+0000 7f43970c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:56.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.256+0000 7f43970c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:21:56.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.256+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43901d7760 con 0x7f43900a98f0 2024-03-25T19:21:56.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.295+0000 7f43857fa640 1 -- 172.21.15.60:0/1326592037 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f438c0523a0 con 0x7f43900a98f0 2024-03-25T19:21:56.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.295+0000 7f43857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:21:56.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.295+0000 7f43857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:21:56.296 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:21:56.296 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.297+0000 7f43970c9640 10 librados: watch_flush enter 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 10 librados: watch_flush exit 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 >> v1:172.21.15.90:6800/1827638754 conn(0x7f437c06ddf0 legacy=0x7f437c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 10 monclient: shutdown 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 >> v1:172.21.15.90:6789/0 conn(0x7f43900a98f0 legacy=0x7f43901d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 shutdown_connections 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 >> 172.21.15.60:0/1326592037 conn(0x7f43900b7ea0 msgr2=0x7f4390074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 shutdown_connections 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 1 -- 172.21.15.60:0/1326592037 wait complete. 2024-03-25T19:21:56.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:21:56.298+0000 7f43970c9640 1 librados: shutdown 2024-03-25T19:21:56.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:03.117 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:03.231 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.231+0000 7ff349a93640 10 monclient: get_monmap_and_config 2024-03-25T19:22:03.231 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.231+0000 7ff349a93640 10 monclient: build_initial_monmap 2024-03-25T19:22:03.231 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.231+0000 7ff349a93640 10 monclient: monmap: 2024-03-25T19:22:03.231 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:03.231 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:03.231 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:03.232498+0000 2024-03-25T19:22:03.231 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:03.232498+0000 2024-03-25T19:22:03.232 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:03.232 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:03.232 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:03.232 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:03.232 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:03.232 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.232+0000 7ff349a93640 1 Processor -- start 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.232+0000 7ff349a93640 1 -- start start 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.232+0000 7ff349a93640 10 monclient: init 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 10 monclient(hunting): picked mon.noname-a con 0x7ff3440a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 10 monclient(hunting): picked mon.noname-c con 0x7ff34413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 10 monclient(hunting): picked mon.noname-b con 0x7ff3441430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:03.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff344142520 con 0x7ff3440a98f0 2024-03-25T19:22:03.234 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440b28a0 con 0x7ff34413f710 2024-03-25T19:22:03.234 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440b2a30 con 0x7ff3441430c0 2024-03-25T19:22:03.234 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:03.234 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.233+0000 7ff349a93640 10 monclient(hunting): authenticate will time out at 6450.331628s 2024-03-25T19:22:03.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.243+0000 7ff342ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff34413f710 0x7ff3441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:48722/0 (socket says 172.21.15.60:48722) 2024-03-25T19:22:03.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.243+0000 7ff342ffd640 1 -- 172.21.15.60:0/2387947357 learned_addr learned my addr 172.21.15.60:0/2387947357 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:03.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.254+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132785278 0 0) 0x7ff3440b2a30 con 0x7ff3441430c0 2024-03-25T19:22:03.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.254+0000 7ff3427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:03.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.254+0000 7ff3427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:03.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.254+0000 7ff3427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:03.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.254+0000 7ff3427fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:22:03.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.254+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff32c003520 con 0x7ff3441430c0 2024-03-25T19:22:03.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.257+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755773833 0 0) 0x7ff3440b28a0 con 0x7ff34413f710 2024-03-25T19:22:03.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.257+0000 7ff3427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:03.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.257+0000 7ff3427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:03.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.257+0000 7ff3427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:03.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.257+0000 7ff3427fc640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:22:03.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.257+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440b2a30 con 0x7ff34413f710 2024-03-25T19:22:03.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2789358463 0 0) 0x7ff32c003520 con 0x7ff3441430c0 2024-03-25T19:22:03.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 >> v1:172.21.15.90:6789/0 conn(0x7ff34413f710 legacy=0x7ff3441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:03.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 >> v1:172.21.15.60:6789/0 conn(0x7ff3440a98f0 legacy=0x7ff3440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:03.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:03.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 1 monclient: found mon.noname-b 2024-03-25T19:22:03.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:03.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3440b44c0 con 0x7ff3441430c0 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.262+0000 7ff3427fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff3427fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff3427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff33c001a10 con 0x7ff3441430c0 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff349a93640 5 monclient: authenticate success, global_id 4289 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff349a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff3427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff3427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff3427fc640 10 monclient: dump: 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.264 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.265 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:03.265 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:03.265 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:03.265 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:03.265 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:03.265 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.263+0000 7ff349a93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff33c001c30 con 0x7ff3441430c0 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff3427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff3427fc640 1 -- 172.21.15.60:0/2387947357 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff33c001fc0 con 0x7ff3441430c0 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff349a93640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff349a93640 4 set_mon_vals no callback set 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff3427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff3427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.281+0000 7ff3427fc640 10 monclient: dump: 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.281 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.282 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:03.282 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:03.282 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:03.282 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:03.282 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:03.282 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.282+0000 7ff349a93640 10 monclient: shutdown 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.282+0000 7ff349a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.282+0000 7ff349a93640 1 -- 172.21.15.60:0/2387947357 >> v1:172.21.15.60:6790/0 conn(0x7ff3441430c0 legacy=0x7ff34414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.282+0000 7ff349a93640 1 -- 172.21.15.60:0/2387947357 shutdown_connections 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.282+0000 7ff349a93640 1 -- 172.21.15.60:0/2387947357 >> 172.21.15.60:0/2387947357 conn(0x7ff3440b7ea0 msgr2=0x7ff3440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.282+0000 7ff349a93640 1 -- 172.21.15.60:0/2387947357 shutdown_connections 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.282+0000 7ff349a93640 1 -- 172.21.15.60:0/2387947357 wait complete. 2024-03-25T19:22:03.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 10 monclient: build_initial_monmap 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 10 monclient: monmap: 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 1 librados: starting msgr at 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 1 librados: starting objecter 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 1 Processor -- start 2024-03-25T19:22:03.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 1 -- start start 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 1 librados: setting wanted keys 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 1 librados: calling monclient init 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.283+0000 7ff349a93640 10 monclient: init 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 10 monclient(hunting): picked mon.noname-b con 0x7ff3440a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 10 monclient(hunting): picked mon.noname-a con 0x7ff34413f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 10 monclient(hunting): picked mon.noname-c con 0x7ff3441430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440abc00 con 0x7ff34413f710 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440ac150 con 0x7ff3440a98f0 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3440abe70 con 0x7ff3441430c0 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:03.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.284+0000 7ff349a93640 10 monclient(hunting): authenticate will time out at 6450.382607s 2024-03-25T19:22:03.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.293+0000 7ff342ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff34413f710 0x7ff3440ac570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38856/0 (socket says 172.21.15.60:38856) 2024-03-25T19:22:03.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.293+0000 7ff342ffd640 1 -- 172.21.15.60:0/1037390191 learned_addr learned my addr 172.21.15.60:0/1037390191 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:03.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.298+0000 7ff32bfff640 10 client.?.objecter ms_handle_connect 0x7ff34413f710 2024-03-25T19:22:03.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.298+0000 7ff32bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:03.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.302+0000 7ff32bfff640 10 client.?.objecter ms_handle_connect 0x7ff3440a98f0 2024-03-25T19:22:03.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.302+0000 7ff32bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:03.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.311+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660136581 0 0) 0x7ff3440ac150 con 0x7ff3440a98f0 2024-03-25T19:22:03.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.311+0000 7ff32bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:03.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.311+0000 7ff32bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:03.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.311+0000 7ff32bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:03.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.311+0000 7ff32bfff640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:22:03.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.311+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff330004eb0 con 0x7ff3440a98f0 2024-03-25T19:22:03.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.318+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3684049168 0 0) 0x7ff3440abc00 con 0x7ff34413f710 2024-03-25T19:22:03.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.318+0000 7ff32bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:03.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.318+0000 7ff32bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:03.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.318+0000 7ff32bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:03.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.318+0000 7ff32bfff640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:22:03.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.318+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440ac150 con 0x7ff34413f710 2024-03-25T19:22:03.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.320+0000 7ff32bfff640 10 client.?.objecter ms_handle_connect 0x7ff3441430c0 2024-03-25T19:22:03.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.320+0000 7ff32bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:03.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.331+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389532464 0 0) 0x7ff3440abe70 con 0x7ff3441430c0 2024-03-25T19:22:03.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.332+0000 7ff32bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:22:03.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.332+0000 7ff32bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:03.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.332+0000 7ff32bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:03.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.332+0000 7ff32bfff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:22:03.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.332+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3440abc00 con 0x7ff3441430c0 2024-03-25T19:22:03.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.337+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968743955 0 0) 0x7ff3440ac150 con 0x7ff34413f710 2024-03-25T19:22:03.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.338+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3440abe70 con 0x7ff34413f710 2024-03-25T19:22:03.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.338+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff3380019f0 con 0x7ff34413f710 2024-03-25T19:22:03.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.338+0000 7ff32bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:03.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.338+0000 7ff32bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:03.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.338+0000 7ff32bfff640 10 monclient(hunting): dump: 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.338+0000 7ff32bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.340+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357905205 0 0) 0x7ff330004eb0 con 0x7ff3440a98f0 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.340+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3440ac150 con 0x7ff3440a98f0 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.340+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff334001930 con 0x7ff3440a98f0 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.340+0000 7ff32bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.340+0000 7ff32bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.340+0000 7ff32bfff640 10 monclient(hunting): dump: 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:03.340 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.341 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.341 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:03.341 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:03.341 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:03.341 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:03.341 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:03.341 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.357+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2227960506 0 0) 0x7ff3440abc00 con 0x7ff3441430c0 2024-03-25T19:22:03.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.357+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff330004eb0 con 0x7ff3441430c0 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.357+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff33c001da0 con 0x7ff3441430c0 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.357+0000 7ff32bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.357+0000 7ff32bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.357+0000 7ff32bfff640 10 monclient(hunting): dump: 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:03.357 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.362 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1395338214 0 0) 0x7ff3440abe70 con 0x7ff34413f710 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 >> v1:172.21.15.60:6790/0 conn(0x7ff3441430c0 legacy=0x7ff3441e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 >> v1:172.21.15.90:6789/0 conn(0x7ff3440a98f0 legacy=0x7ff344136580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 1 monclient: found mon.a 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3441390f0 con 0x7ff34413f710 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 10 monclient: _finish_auth 0 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff32bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 5 monclient: authenticate success, global_id 4308 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 10 monclient: _renew_subs 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff344137e00 con 0x7ff34413f710 2024-03-25T19:22:03.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 10 monclient: _renew_subs 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff344138bf0 con 0x7ff34413f710 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.363+0000 7ff349a93640 1 librados: init done 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.364+0000 7ff349a93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.364+0000 7ff349a93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.364+0000 7ff349a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:03.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.364+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3440627b0 con 0x7ff34413f710 2024-03-25T19:22:03.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3380047e0 con 0x7ff34413f710 2024-03-25T19:22:03.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff32bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:03.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff338004af0 con 0x7ff34413f710 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff32bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff32bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff32bfff640 10 monclient: dump: 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:03.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff341ffb640 4 set_mon_vals no callback set 2024-03-25T19:22:03.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.713+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7ff33801aac0 con 0x7ff34413f710 2024-03-25T19:22:03.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.714+0000 7ff3437fe640 10 monclient: get_auth_request con 0x7ff33006e0b0 auth_method 0 2024-03-25T19:22:03.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.714+0000 7ff32bfff640 10 client.4308.objecter ms_handle_connect 0x7ff33006e0b0 2024-03-25T19:22:03.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7ff338054810 con 0x7ff34413f710 2024-03-25T19:22:03.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 10 client.4308.objecter ms_dispatch 0x7ff344000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:03.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:03.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:03.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-25T19:22:03.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff338050150 con 0x7ff34413f710 2024-03-25T19:22:03.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:03.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.720+0000 7ff32bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:03.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.957+0000 7ff349a93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:03.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.957+0000 7ff349a93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:03.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.957+0000 7ff349a93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:03.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:03.957+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff344141bb0 con 0x7ff34413f710 2024-03-25T19:22:04.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.000+0000 7ff32bfff640 1 -- 172.21.15.60:0/1037390191 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7ff338052250 con 0x7ff34413f710 2024-03-25T19:22:04.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.000+0000 7ff32bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:04.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.000+0000 7ff32bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:04.000 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:04.000 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:04.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 10 librados: watch_flush enter 2024-03-25T19:22:04.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 10 librados: watch_flush exit 2024-03-25T19:22:04.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-25T19:22:04.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 10 client.4308.objecter successfully canceled tick 2024-03-25T19:22:04.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 >> v1:172.21.15.90:6800/1827638754 conn(0x7ff33006e0b0 legacy=0x7ff330070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 10 monclient: shutdown 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 >> v1:172.21.15.60:6789/0 conn(0x7ff34413f710 legacy=0x7ff3440ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 shutdown_connections 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.003+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 >> 172.21.15.60:0/1037390191 conn(0x7ff3440b7ea0 msgr2=0x7ff344074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.004+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 shutdown_connections 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.004+0000 7ff349a93640 1 -- 172.21.15.60:0/1037390191 wait complete. 2024-03-25T19:22:04.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:04.004+0000 7ff349a93640 1 librados: shutdown 2024-03-25T19:22:04.015 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:10.821 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.933+0000 7f5d7e825640 10 monclient: get_monmap_and_config 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.933+0000 7f5d7e825640 10 monclient: build_initial_monmap 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.933+0000 7f5d7e825640 10 monclient: monmap: 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:10.934475+0000 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:10.934475+0000 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:10.933 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:10.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.934+0000 7f5d7e825640 1 Processor -- start 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.934+0000 7f5d7e825640 1 -- start start 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.934+0000 7f5d7e825640 10 monclient: init 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 10 monclient(hunting): picked mon.noname-c con 0x7f5d780a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 10 monclient(hunting): picked mon.noname-b con 0x7f5d781430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d78142520 con 0x7f5d7813f710 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d780b28a0 con 0x7f5d780a98f0 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d780b2a30 con 0x7f5d781430c0 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:10.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.935+0000 7f5d7e825640 10 monclient(hunting): authenticate will time out at 6458.033307s 2024-03-25T19:22:10.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.938+0000 7f5d777fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f5d7813f710 0x7f5d781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:42262/0 (socket says 172.21.15.60:42262) 2024-03-25T19:22:10.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.938+0000 7f5d777fe640 1 -- 172.21.15.60:0/3360869748 learned_addr learned my addr 172.21.15.60:0/3360869748 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:10.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.963+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134995046 0 0) 0x7f5d78142520 con 0x7f5d7813f710 2024-03-25T19:22:10.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.963+0000 7f5d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:10.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.963+0000 7f5d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:10.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.963+0000 7f5d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:10.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.963+0000 7f5d76ffd640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:22:10.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.963+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d60003520 con 0x7f5d7813f710 2024-03-25T19:22:10.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.970+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497792505 0 0) 0x7f5d780b28a0 con 0x7f5d780a98f0 2024-03-25T19:22:10.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.970+0000 7f5d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:10.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.970+0000 7f5d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:10.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.970+0000 7f5d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:10.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.970+0000 7f5d76ffd640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:22:10.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.970+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d78142520 con 0x7f5d780a98f0 2024-03-25T19:22:10.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.975+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786660646 0 0) 0x7f5d780b2a30 con 0x7f5d781430c0 2024-03-25T19:22:10.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.975+0000 7f5d76ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:10.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.975+0000 7f5d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:10.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.975+0000 7f5d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:10.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.975+0000 7f5d76ffd640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:22:10.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.975+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d780b28a0 con 0x7f5d781430c0 2024-03-25T19:22:10.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 958414417 0 0) 0x7f5d60003520 con 0x7f5d7813f710 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 >> v1:172.21.15.60:6790/0 conn(0x7f5d781430c0 legacy=0x7f5d7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 >> v1:172.21.15.90:6789/0 conn(0x7f5d780a98f0 legacy=0x7f5d780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 1 monclient: found mon.noname-a 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d780b44c0 con 0x7f5d7813f710 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5d6c001690 con 0x7f5d7813f710 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d7e825640 5 monclient: authenticate success, global_id 4311 2024-03-25T19:22:10.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d7e825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d76ffd640 10 monclient: dump: 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:10.986 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:10.987 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:10.987 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:10.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.985+0000 7f5d7e825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d6c0017f0 con 0x7f5d7813f710 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d76ffd640 1 -- 172.21.15.60:0/3360869748 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5d6c001c00 con 0x7f5d7813f710 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d76ffd640 10 monclient: dump: 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:10.998 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:10.999 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:10.999 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:10.999 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:10.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d7e825640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:10.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.998+0000 7f5d7e825640 4 set_mon_vals no callback set 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.999+0000 7f5d7e825640 10 monclient: shutdown 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.999+0000 7f5d7e825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.999+0000 7f5d7e825640 1 -- 172.21.15.60:0/3360869748 >> v1:172.21.15.60:6789/0 conn(0x7f5d7813f710 legacy=0x7f5d781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.999+0000 7f5d7e825640 1 -- 172.21.15.60:0/3360869748 shutdown_connections 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.999+0000 7f5d7e825640 1 -- 172.21.15.60:0/3360869748 >> 172.21.15.60:0/3360869748 conn(0x7f5d780b7ea0 msgr2=0x7f5d780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.999+0000 7f5d7e825640 1 -- 172.21.15.60:0/3360869748 shutdown_connections 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:10.999+0000 7f5d7e825640 1 -- 172.21.15.60:0/3360869748 wait complete. 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 10 monclient: build_initial_monmap 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 10 monclient: monmap: 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:11.001 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 1 librados: starting msgr at 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 1 librados: starting objecter 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 1 Processor -- start 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 1 -- start start 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 1 librados: setting wanted keys 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 1 librados: calling monclient init 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.000+0000 7f5d7e825640 10 monclient: init 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:11.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 10 monclient(hunting): picked mon.noname-c con 0x7f5d780a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 10 monclient(hunting): picked mon.noname-a con 0x7f5d7813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 10 monclient(hunting): picked mon.noname-b con 0x7f5d781430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d780abac0 con 0x7f5d7813f710 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d780abda0 con 0x7f5d781430c0 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d781380e0 con 0x7f5d780a98f0 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:11.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.001+0000 7f5d7e825640 10 monclient(hunting): authenticate will time out at 6458.099644s 2024-03-25T19:22:11.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.013+0000 7f5d77fff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f5d780a98f0 0x7f5d780ac580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:47486/0 (socket says 172.21.15.60:47486) 2024-03-25T19:22:11.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.013+0000 7f5d77fff640 1 -- 172.21.15.60:0/3277627333 learned_addr learned my addr 172.21.15.60:0/3277627333 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:11.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.016+0000 7f5d74ff9640 10 client.?.objecter ms_handle_connect 0x7f5d780a98f0 2024-03-25T19:22:11.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.016+0000 7f5d74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:11.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.020+0000 7f5d74ff9640 10 client.?.objecter ms_handle_connect 0x7f5d7813f710 2024-03-25T19:22:11.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.020+0000 7f5d74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:11.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.022+0000 7f5d74ff9640 10 client.?.objecter ms_handle_connect 0x7f5d781430c0 2024-03-25T19:22:11.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.022+0000 7f5d74ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:11.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.025+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972536513 0 0) 0x7f5d781380e0 con 0x7f5d780a98f0 2024-03-25T19:22:11.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.025+0000 7f5d74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:11.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.025+0000 7f5d74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:11.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.025+0000 7f5d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:11.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.025+0000 7f5d74ff9640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:22:11.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.025+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d64004eb0 con 0x7f5d780a98f0 2024-03-25T19:22:11.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.041+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650061552 0 0) 0x7f5d780abac0 con 0x7f5d7813f710 2024-03-25T19:22:11.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.041+0000 7f5d74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:11.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.041+0000 7f5d74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:11.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.041+0000 7f5d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:11.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.041+0000 7f5d74ff9640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:22:11.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.041+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d781380e0 con 0x7f5d7813f710 2024-03-25T19:22:11.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.046+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443776208 0 0) 0x7f5d780abda0 con 0x7f5d781430c0 2024-03-25T19:22:11.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.047+0000 7f5d74ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:11.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.047+0000 7f5d74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:11.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.047+0000 7f5d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:11.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.047+0000 7f5d74ff9640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:22:11.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.047+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d780abac0 con 0x7f5d781430c0 2024-03-25T19:22:11.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.049+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064513947 0 0) 0x7f5d64004eb0 con 0x7f5d780a98f0 2024-03-25T19:22:11.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.049+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d780abda0 con 0x7f5d780a98f0 2024-03-25T19:22:11.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.049+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5d68001a30 con 0x7f5d780a98f0 2024-03-25T19:22:11.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.049+0000 7f5d74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:11.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.049+0000 7f5d74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:11.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.049+0000 7f5d74ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:11.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.050+0000 7f5d74ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:11.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.054+0000 7f5d777fe640 1 -- 172.21.15.60:0/3277627333 >> v1:172.21.15.60:6789/0 conn(0x7f5d7813f710 legacy=0x7f5d78138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T19:22:11.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.054+0000 7f5d777fe640 1 -- 172.21.15.60:0/3277627333 >> v1:172.21.15.60:6789/0 conn(0x7f5d7813f710 legacy=0x7f5d78138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:22:11.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.054+0000 7f5d777fe640 1 --1- 172.21.15.60:0/3277627333 >> v1:172.21.15.60:6789/0 conn(0x7f5d7813f710 0x7f5d78138b30 :-1 s=OPENED pgs=9729 cs=1 l=1).handle_message read tag failed 2024-03-25T19:22:11.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.054+0000 7f5d777fe640 1 --1- 172.21.15.60:0/3277627333 >> v1:172.21.15.60:6789/0 conn(0x7f5d7813f710 0x7f5d78138b30 :-1 s=OPENED pgs=9729 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:22:11.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.054+0000 7f5d74ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.60:6789/0 2024-03-25T19:22:11.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3878290625 0 0) 0x7f5d780abda0 con 0x7f5d780a98f0 2024-03-25T19:22:11.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 >> v1:172.21.15.90:6789/0 conn(0x7f5d781430c0 legacy=0x7f5d781e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:11.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 >> v1:172.21.15.60:6789/0 conn(0x7f5d7813f710 legacy=0x7f5d78138b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:22:11.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:22:11.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 1 monclient: found mon.c 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d78137d90 con 0x7f5d780a98f0 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 10 monclient: _finish_auth 0 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 5 monclient: authenticate success, global_id 4298 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 10 monclient: _renew_subs 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d78139240 con 0x7f5d780a98f0 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 10 monclient: _renew_subs 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:22:11.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d78136830 con 0x7f5d780a98f0 2024-03-25T19:22:11.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.061+0000 7f5d7e825640 1 librados: init done 2024-03-25T19:22:11.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.062+0000 7f5d7e825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:11.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.062+0000 7f5d7e825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:11.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.062+0000 7f5d7e825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:22:11.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.062+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d780627b0 con 0x7f5d780a98f0 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.401+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d68001c70 con 0x7f5d780a98f0 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.401+0000 7f5d74ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.401+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5d68003a60 con 0x7f5d780a98f0 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.401+0000 7f5d74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.401+0000 7f5d74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.401+0000 7f5d74ff9640 10 monclient: dump: 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:11.401 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.401+0000 7f5d767fc640 4 set_mon_vals no callback set 2024-03-25T19:22:11.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.402+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f5d68003c40 con 0x7f5d780a98f0 2024-03-25T19:22:11.403 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.402+0000 7f5d777fe640 10 monclient: get_auth_request con 0x7f5d6406ddf0 auth_method 0 2024-03-25T19:22:11.403 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.402+0000 7f5d74ff9640 10 client.4298.objecter ms_handle_connect 0x7f5d6406ddf0 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f5d680548d0 con 0x7f5d780a98f0 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 10 client.4298.objecter ms_dispatch 0x7f5d78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d680503e0 con 0x7f5d780a98f0 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:11.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.407+0000 7f5d74ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:11.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.667+0000 7f5d7e825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:11.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.667+0000 7f5d7e825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:11.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.667+0000 7f5d7e825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:22:11.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.667+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d7813f070 con 0x7f5d780a98f0 2024-03-25T19:22:11.845 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.845+0000 7f5d74ff9640 1 -- 172.21.15.60:0/3277627333 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f5d68052310 con 0x7f5d780a98f0 2024-03-25T19:22:11.845 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.845+0000 7f5d74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:11.845 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.845+0000 7f5d74ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:11.845 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:11.845 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:11.848 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 10 librados: watch_flush enter 2024-03-25T19:22:11.848 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 10 librados: watch_flush exit 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 10 client.4298.objecter successfully canceled tick 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 >> v1:172.21.15.90:6800/1827638754 conn(0x7f5d6406ddf0 legacy=0x7f5d640702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 10 monclient: shutdown 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 >> v1:172.21.15.60:6790/0 conn(0x7f5d780a98f0 legacy=0x7f5d780ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 shutdown_connections 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 >> 172.21.15.60:0/3277627333 conn(0x7f5d780b7ea0 msgr2=0x7f5d78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 shutdown_connections 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 1 -- 172.21.15.60:0/3277627333 wait complete. 2024-03-25T19:22:11.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:11.848+0000 7f5d7e825640 1 librados: shutdown 2024-03-25T19:22:11.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:13.932 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:14.643 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:13.976 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:14.630 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:13.981 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:14.635 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:13.988 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:14.699 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:14.021 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:14.675 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:14.031 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:14.742 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:14.037 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:14.749 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:14.063 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:14.774 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:14.084 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:14.796 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:18.666 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:18.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.785+0000 7fe277e33640 10 monclient: get_monmap_and_config 2024-03-25T19:22:18.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.785+0000 7fe277e33640 10 monclient: build_initial_monmap 2024-03-25T19:22:18.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.785+0000 7fe277e33640 10 monclient: monmap: 2024-03-25T19:22:18.785 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:18.785 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:18.785 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:18.786434+0000 2024-03-25T19:22:18.786 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:18.786434+0000 2024-03-25T19:22:18.786 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:18.786 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:18.786 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:18.786 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:18.786 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:18.786 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:18.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.786+0000 7fe277e33640 1 Processor -- start 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 1 -- start start 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 10 monclient: init 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 10 monclient(hunting): picked mon.noname-a con 0x7fe2700a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 10 monclient(hunting): picked mon.noname-c con 0x7fe27013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 10 monclient(hunting): picked mon.noname-b con 0x7fe2701430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:18.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe270142520 con 0x7fe2700a98f0 2024-03-25T19:22:18.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.787+0000 7fe277e33640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700b28a0 con 0x7fe27013f710 2024-03-25T19:22:18.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.788+0000 7fe277e33640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700b2a30 con 0x7fe2701430c0 2024-03-25T19:22:18.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.788+0000 7fe277e33640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:18.789 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.788+0000 7fe277e33640 10 monclient(hunting): authenticate will time out at 6465.885901s 2024-03-25T19:22:18.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.792+0000 7fe2763d0640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fe2701430c0 0x7fe27014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:40100/0 (socket says 172.21.15.60:40100) 2024-03-25T19:22:18.792 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.792+0000 7fe2763d0640 1 -- 172.21.15.60:0/73391636 learned_addr learned my addr 172.21.15.60:0/73391636 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:18.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.801+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510212369 0 0) 0x7fe2700b28a0 con 0x7fe27013f710 2024-03-25T19:22:18.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.802+0000 7fe274bcd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:18.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.802+0000 7fe274bcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:18.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.802+0000 7fe274bcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:18.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.802+0000 7fe274bcd640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:22:18.802 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.802+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe25c003520 con 0x7fe27013f710 2024-03-25T19:22:18.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.804+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369450829 0 0) 0x7fe2700b2a30 con 0x7fe2701430c0 2024-03-25T19:22:18.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.804+0000 7fe274bcd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:18.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.804+0000 7fe274bcd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:18.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.804+0000 7fe274bcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:18.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.804+0000 7fe274bcd640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:22:18.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.804+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2700b28a0 con 0x7fe2701430c0 2024-03-25T19:22:18.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.808+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500751374 0 0) 0x7fe25c003520 con 0x7fe27013f710 2024-03-25T19:22:18.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.808+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 >> v1:172.21.15.60:6790/0 conn(0x7fe2701430c0 legacy=0x7fe27014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:18.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.808+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 >> v1:172.21.15.60:6789/0 conn(0x7fe2700a98f0 legacy=0x7fe2700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:22:18.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:18.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 1 monclient: found mon.noname-c 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2700b44c0 con 0x7fe27013f710 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 10 monclient: _finish_auth 0 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe274bcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe277e33640 5 monclient: authenticate success, global_id 4303 2024-03-25T19:22:18.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.809+0000 7fe277e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe260001650 con 0x7fe27013f710 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 10 monclient: dump: 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.817 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2600018a0 con 0x7fe27013f710 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe277e33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe277e33640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe277e33640 4 set_mon_vals no callback set 2024-03-25T19:22:18.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 1 -- 172.21.15.60:0/73391636 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe260001c30 con 0x7fe27013f710 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.817+0000 7fe274bcd640 10 monclient: dump: 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:18.819 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:18.820 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:18.820 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:18.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.818+0000 7fe277e33640 10 monclient: shutdown 2024-03-25T19:22:18.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.818+0000 7fe277e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:18.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.818+0000 7fe277e33640 1 -- 172.21.15.60:0/73391636 >> v1:172.21.15.90:6789/0 conn(0x7fe27013f710 legacy=0x7fe2701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:18.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.818+0000 7fe277e33640 1 -- 172.21.15.60:0/73391636 shutdown_connections 2024-03-25T19:22:18.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.818+0000 7fe277e33640 1 -- 172.21.15.60:0/73391636 >> 172.21.15.60:0/73391636 conn(0x7fe2700b7ea0 msgr2=0x7fe2700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:18.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.818+0000 7fe277e33640 1 -- 172.21.15.60:0/73391636 shutdown_connections 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.818+0000 7fe277e33640 1 -- 172.21.15.60:0/73391636 wait complete. 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.819+0000 7fe277e33640 10 monclient: build_initial_monmap 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.819+0000 7fe277e33640 10 monclient: monmap: 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:18.823 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 1 librados: starting msgr at 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 1 librados: starting objecter 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 1 Processor -- start 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 1 -- start start 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 1 librados: setting wanted keys 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 1 librados: calling monclient init 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 10 monclient: init 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:18.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 10 monclient(hunting): picked mon.noname-c con 0x7fe2700a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:18.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.820+0000 7fe277e33640 10 monclient(hunting): picked mon.noname-a con 0x7fe27013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:18.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.821+0000 7fe277e33640 10 monclient(hunting): picked mon.noname-b con 0x7fe2701430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:18.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.821+0000 7fe277e33640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700b1730 con 0x7fe27013f710 2024-03-25T19:22:18.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.821+0000 7fe277e33640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700b0920 con 0x7fe2701430c0 2024-03-25T19:22:18.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.821+0000 7fe277e33640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2701dae10 con 0x7fe2700a98f0 2024-03-25T19:22:18.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.821+0000 7fe277e33640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:18.826 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.821+0000 7fe277e33640 10 monclient(hunting): authenticate will time out at 6465.918988s 2024-03-25T19:22:18.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.832+0000 7fe275bcf640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fe2700a98f0 0x7fe2701d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:40130/0 (socket says 172.21.15.60:40130) 2024-03-25T19:22:18.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.832+0000 7fe275bcf640 1 -- 172.21.15.60:0/2695070416 learned_addr learned my addr 172.21.15.60:0/2695070416 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:18.835 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.835+0000 7fe2667fc640 10 client.?.objecter ms_handle_connect 0x7fe2700a98f0 2024-03-25T19:22:18.835 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.835+0000 7fe2667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:18.836 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.836+0000 7fe2667fc640 10 client.?.objecter ms_handle_connect 0x7fe27013f710 2024-03-25T19:22:18.836 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.836+0000 7fe2667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:18.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.843+0000 7fe2667fc640 10 client.?.objecter ms_handle_connect 0x7fe2701430c0 2024-03-25T19:22:18.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.843+0000 7fe2667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:18.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.844+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409665483 0 0) 0x7fe2701dae10 con 0x7fe2700a98f0 2024-03-25T19:22:18.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.844+0000 7fe2667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:18.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.844+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:18.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.844+0000 7fe2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:18.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.844+0000 7fe2667fc640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:22:18.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.844+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe254004eb0 con 0x7fe2700a98f0 2024-03-25T19:22:18.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.855+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764145510 0 0) 0x7fe2700b0920 con 0x7fe2701430c0 2024-03-25T19:22:18.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.855+0000 7fe2667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:18.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.855+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:18.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.855+0000 7fe2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:18.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.855+0000 7fe2667fc640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:22:18.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.855+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2701dae10 con 0x7fe2701430c0 2024-03-25T19:22:18.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.859+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2412445002 0 0) 0x7fe254004eb0 con 0x7fe2700a98f0 2024-03-25T19:22:18.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.859+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2700b0920 con 0x7fe2700a98f0 2024-03-25T19:22:18.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.859+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe26c001960 con 0x7fe2700a98f0 2024-03-25T19:22:18.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.859+0000 7fe2667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:18.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.859+0000 7fe2667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:18.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.859+0000 7fe2667fc640 10 monclient(hunting): dump: 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:18.861 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:18.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.859+0000 7fe2667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:18.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.862+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1509602273 0 0) 0x7fe2700b1730 con 0x7fe27013f710 2024-03-25T19:22:18.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.862+0000 7fe2667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:18.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.862+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:18.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.862+0000 7fe2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:18.863 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.862+0000 7fe2667fc640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:22:18.863 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.862+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe254004eb0 con 0x7fe27013f710 2024-03-25T19:22:18.863 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.864+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 813197295 0 0) 0x7fe2701dae10 con 0x7fe2701430c0 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.864+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2700b1730 con 0x7fe2701430c0 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.864+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe268001b50 con 0x7fe2701430c0 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.864+0000 7fe2667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.864+0000 7fe2667fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.864+0000 7fe2667fc640 10 monclient(hunting): dump: 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:18.864 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:18.865 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3687422581 0 0) 0x7fe2700b1730 con 0x7fe2701430c0 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 >> v1:172.21.15.60:6790/0 conn(0x7fe2700a98f0 legacy=0x7fe2701d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 >> v1:172.21.15.60:6789/0 conn(0x7fe27013f710 legacy=0x7fe2700acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 1 monclient: found mon.b 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2701d81b0 con 0x7fe2701430c0 2024-03-25T19:22:18.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe2667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 5 monclient: authenticate success, global_id 4306 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 10 monclient: _renew_subs 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2701da8f0 con 0x7fe2701430c0 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 10 monclient: _renew_subs 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27014dfa0 con 0x7fe2701430c0 2024-03-25T19:22:18.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.870+0000 7fe277e33640 1 librados: init done 2024-03-25T19:22:18.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.871+0000 7fe277e33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:18.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.871+0000 7fe277e33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:18.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.871+0000 7fe277e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:18.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:18.871+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2701dafa0 con 0x7fe2701430c0 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.160+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe268002000 con 0x7fe2701430c0 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.160+0000 7fe2667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.160+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe268004d00 con 0x7fe2701430c0 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.160+0000 7fe2667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.160+0000 7fe2667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.160+0000 7fe2667fc640 10 monclient: dump: 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:19.160 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:19.161 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:19.161 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:19.161 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:19.161 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:19.161 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:19.161 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.160+0000 7fe267fff640 4 set_mon_vals no callback set 2024-03-25T19:22:19.161 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.161+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fe26801acd0 con 0x7fe2701430c0 2024-03-25T19:22:19.162 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.162+0000 7fe275bcf640 10 monclient: get_auth_request con 0x7fe25406df50 auth_method 0 2024-03-25T19:22:19.162 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.162+0000 7fe2667fc640 10 client.4306.objecter ms_handle_connect 0x7fe25406df50 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fe268054980 con 0x7fe2701430c0 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 10 client.4306.objecter ms_dispatch 0x7fe270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2680502c0 con 0x7fe2701430c0 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:19.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.166+0000 7fe2667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:19.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.425+0000 7fe277e33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:19.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.425+0000 7fe277e33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:19.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.425+0000 7fe277e33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:19.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.425+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2701d8340 con 0x7fe2701430c0 2024-03-25T19:22:19.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.446+0000 7fe2667fc640 1 -- 172.21.15.60:0/2695070416 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fe2701d8340 con 0x7fe2701430c0 2024-03-25T19:22:19.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.446+0000 7fe2667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:19.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.446+0000 7fe2667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:19.446 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:19.446 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 10 librados: watch_flush enter 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 10 librados: watch_flush exit 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 10 client.4306.objecter successfully canceled tick 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 >> v1:172.21.15.90:6800/1827638754 conn(0x7fe25406df50 legacy=0x7fe254070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 10 monclient: shutdown 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.448+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 >> v1:172.21.15.90:6789/0 conn(0x7fe2701430c0 legacy=0x7fe2700b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.449+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 shutdown_connections 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.449+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 >> 172.21.15.60:0/2695070416 conn(0x7fe2700b7ea0 msgr2=0x7fe270074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.449+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 shutdown_connections 2024-03-25T19:22:19.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.449+0000 7fe277e33640 1 -- 172.21.15.60:0/2695070416 wait complete. 2024-03-25T19:22:19.450 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:19.449+0000 7fe277e33640 1 librados: shutdown 2024-03-25T19:22:19.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:26.265 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.377+0000 7f63dcc4e640 10 monclient: get_monmap_and_config 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.377+0000 7f63dcc4e640 10 monclient: build_initial_monmap 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.377+0000 7f63dcc4e640 10 monclient: monmap: 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:26.378430+0000 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:26.378430+0000 2024-03-25T19:22:26.377 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:26.378 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:26.378 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:26.378 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:26.378 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:26.378 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.378+0000 7f63dcc4e640 1 Processor -- start 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.378+0000 7f63dcc4e640 1 -- start start 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.378+0000 7f63dcc4e640 10 monclient: init 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 10 monclient(hunting): picked mon.noname-c con 0x7f63d80a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 10 monclient(hunting): picked mon.noname-b con 0x7f63d813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 10 monclient(hunting): picked mon.noname-a con 0x7f63d81430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:26.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8142520 con 0x7f63d81430c0 2024-03-25T19:22:26.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80b28a0 con 0x7f63d80a98f0 2024-03-25T19:22:26.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80b2a30 con 0x7f63d813f710 2024-03-25T19:22:26.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:26.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.379+0000 7f63dcc4e640 10 monclient(hunting): authenticate will time out at 6473.477478s 2024-03-25T19:22:26.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.381+0000 7f63d659c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f63d80a98f0 0x7f63d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:59266/0 (socket says 172.21.15.60:59266) 2024-03-25T19:22:26.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.381+0000 7f63d659c640 1 -- 172.21.15.60:0/3618232994 learned_addr learned my addr 172.21.15.60:0/3618232994 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:26.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.396+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360209995 0 0) 0x7f63d80b28a0 con 0x7f63d80a98f0 2024-03-25T19:22:26.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.396+0000 7f63d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:22:26.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.396+0000 7f63d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:26.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.396+0000 7f63d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:26.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.396+0000 7f63d5d9b640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:22:26.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.396+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4003520 con 0x7f63d80a98f0 2024-03-25T19:22:26.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.402+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174645250 0 0) 0x7f63d80b2a30 con 0x7f63d813f710 2024-03-25T19:22:26.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.403+0000 7f63d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:22:26.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.403+0000 7f63d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:26.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.403+0000 7f63d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:26.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.403+0000 7f63d5d9b640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:22:26.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.403+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d80b28a0 con 0x7f63d813f710 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3001429312 0 0) 0x7f63c4003520 con 0x7f63d80a98f0 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 >> v1:172.21.15.60:6790/0 conn(0x7f63d813f710 legacy=0x7f63d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 >> v1:172.21.15.60:6789/0 conn(0x7f63d81430c0 legacy=0x7f63d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 1 monclient: found mon.noname-c 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d80b44c0 con 0x7f63d80a98f0 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:22:26.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f63c8001690 con 0x7f63d80a98f0 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63d5d9b640 10 monclient: dump: 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:26.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63dcc4e640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:22:26.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.407+0000 7f63dcc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:26.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63c80017f0 con 0x7f63d80a98f0 2024-03-25T19:22:26.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:26.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63d5d9b640 1 -- 172.21.15.60:0/3618232994 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f63c8001c00 con 0x7f63d80a98f0 2024-03-25T19:22:26.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63d5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63d5d9b640 10 monclient: dump: 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63dcc4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:26.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63dcc4e640 4 set_mon_vals no callback set 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63dcc4e640 10 monclient: shutdown 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63dcc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.411+0000 7f63dcc4e640 1 -- 172.21.15.60:0/3618232994 >> v1:172.21.15.90:6789/0 conn(0x7f63d80a98f0 legacy=0x7f63d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.412+0000 7f63dcc4e640 1 -- 172.21.15.60:0/3618232994 shutdown_connections 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.412+0000 7f63dcc4e640 1 -- 172.21.15.60:0/3618232994 >> 172.21.15.60:0/3618232994 conn(0x7f63d80b7ea0 msgr2=0x7f63d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.412+0000 7f63dcc4e640 1 -- 172.21.15.60:0/3618232994 shutdown_connections 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.412+0000 7f63dcc4e640 1 -- 172.21.15.60:0/3618232994 wait complete. 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 10 monclient: build_initial_monmap 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 10 monclient: monmap: 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:26.413 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 1 librados: starting msgr at 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 1 librados: starting objecter 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 1 Processor -- start 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 1 -- start start 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 1 librados: setting wanted keys 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 1 librados: calling monclient init 2024-03-25T19:22:26.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.413+0000 7f63dcc4e640 10 monclient: init 2024-03-25T19:22:26.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:26.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:22:26.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 10 monclient(hunting): picked mon.noname-b con 0x7f63d80a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:26.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 10 monclient(hunting): picked mon.noname-a con 0x7f63d813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:26.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 10 monclient(hunting): picked mon.noname-c con 0x7f63d81430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:26.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8135730 con 0x7f63d813f710 2024-03-25T19:22:26.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d81358c0 con 0x7f63d80a98f0 2024-03-25T19:22:26.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8135a50 con 0x7f63d81430c0 2024-03-25T19:22:26.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:26.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.414+0000 7f63dcc4e640 10 monclient(hunting): authenticate will time out at 6473.512361s 2024-03-25T19:22:26.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.417+0000 7f63d659c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f63d80a98f0 0x7f63d81dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:59282/0 (socket says 172.21.15.60:59282) 2024-03-25T19:22:26.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.417+0000 7f63d659c640 1 -- 172.21.15.60:0/2546160643 learned_addr learned my addr 172.21.15.60:0/2546160643 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:26.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.418+0000 7f63ceffd640 10 client.?.objecter ms_handle_connect 0x7f63d80a98f0 2024-03-25T19:22:26.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.418+0000 7f63ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:26.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.423+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149939304 0 0) 0x7f63d81358c0 con 0x7f63d80a98f0 2024-03-25T19:22:26.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.423+0000 7f63ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:26.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.423+0000 7f63ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:26.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.423+0000 7f63ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:26.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.423+0000 7f63ceffd640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:22:26.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.423+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b8004eb0 con 0x7f63d80a98f0 2024-03-25T19:22:26.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.425+0000 7f63ceffd640 10 client.?.objecter ms_handle_connect 0x7f63d813f710 2024-03-25T19:22:26.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.425+0000 7f63ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:26.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.435+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90187175 0 0) 0x7f63d8135730 con 0x7f63d813f710 2024-03-25T19:22:26.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.435+0000 7f63ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:26.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.435+0000 7f63ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:26.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.435+0000 7f63ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:26.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.435+0000 7f63ceffd640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:22:26.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.435+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d81358c0 con 0x7f63d813f710 2024-03-25T19:22:26.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.436+0000 7f63ceffd640 10 client.?.objecter ms_handle_connect 0x7f63d81430c0 2024-03-25T19:22:26.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.436+0000 7f63ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.437+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1839594699 0 0) 0x7f63b8004eb0 con 0x7f63d80a98f0 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.437+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63d8135730 con 0x7f63d80a98f0 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.437+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f63c8001c40 con 0x7f63d80a98f0 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.437+0000 7f63ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.437+0000 7f63ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.437+0000 7f63ceffd640 10 monclient(hunting): dump: 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:26.437 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:26.438 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:26.438 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:26.438 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:26.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.437+0000 7f63ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:26.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.456+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947303988 0 0) 0x7f63d8135a50 con 0x7f63d81430c0 2024-03-25T19:22:26.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.456+0000 7f63ceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:26.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.456+0000 7f63ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:26.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.456+0000 7f63ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:26.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.456+0000 7f63ceffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:22:26.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.456+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b8004eb0 con 0x7f63d81430c0 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4211480898 0 0) 0x7f63d8135730 con 0x7f63d80a98f0 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 >> v1:172.21.15.60:6790/0 conn(0x7f63d81430c0 legacy=0x7f63d81d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 >> v1:172.21.15.60:6789/0 conn(0x7f63d813f710 legacy=0x7f63d81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 1 monclient: found mon.b 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d81d5000 con 0x7f63d80a98f0 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:26.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 5 monclient: authenticate success, global_id 4312 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 10 monclient: _renew_subs 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d81d52b0 con 0x7f63d80a98f0 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 10 monclient: _renew_subs 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d813c900 con 0x7f63d80a98f0 2024-03-25T19:22:26.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.459+0000 7f63dcc4e640 1 librados: init done 2024-03-25T19:22:26.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.460+0000 7f63dcc4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:26.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.460+0000 7f63dcc4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:26.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.460+0000 7f63dcc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:26.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.460+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d813cbb0 con 0x7f63d80a98f0 2024-03-25T19:22:26.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.726+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63c8004a90 con 0x7f63d80a98f0 2024-03-25T19:22:26.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.727+0000 7f63ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:26.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.727+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f63c8004c40 con 0x7f63d80a98f0 2024-03-25T19:22:26.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.727+0000 7f63ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.727+0000 7f63ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.727+0000 7f63ceffd640 10 monclient: dump: 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:26.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.727+0000 7f63d559a640 4 set_mon_vals no callback set 2024-03-25T19:22:26.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.728+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f63c8004e60 con 0x7f63d80a98f0 2024-03-25T19:22:26.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.728+0000 7f63cffff640 10 monclient: get_auth_request con 0x7f63b806de00 auth_method 0 2024-03-25T19:22:26.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.728+0000 7f63ceffd640 10 client.4312.objecter ms_handle_connect 0x7f63b806de00 2024-03-25T19:22:26.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.733+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f63c80547b0 con 0x7f63d80a98f0 2024-03-25T19:22:26.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.733+0000 7f63ceffd640 10 client.4312.objecter ms_dispatch 0x7f63d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:26.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.733+0000 7f63ceffd640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:26.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.733+0000 7f63ceffd640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:26.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.733+0000 7f63ceffd640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T19:22:26.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.734+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63c80502c0 con 0x7f63d80a98f0 2024-03-25T19:22:26.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.735+0000 7f63ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:26.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.735+0000 7f63ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:26.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.992+0000 7f63dcc4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:26.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.993+0000 7f63dcc4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:26.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.993+0000 7f63dcc4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:26.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:26.993+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63d81371c0 con 0x7f63d80a98f0 2024-03-25T19:22:27.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.061+0000 7f63ceffd640 1 -- 172.21.15.60:0/2546160643 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f63c80521f0 con 0x7f63d80a98f0 2024-03-25T19:22:27.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.061+0000 7f63ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:27.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.061+0000 7f63ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:27.061 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:27.061 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:27.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 10 librados: watch_flush enter 2024-03-25T19:22:27.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 10 librados: watch_flush exit 2024-03-25T19:22:27.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T19:22:27.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 10 client.4312.objecter successfully canceled tick 2024-03-25T19:22:27.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 >> v1:172.21.15.90:6800/1827638754 conn(0x7f63b806de00 legacy=0x7f63b80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:27.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 10 monclient: shutdown 2024-03-25T19:22:27.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:27.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 >> v1:172.21.15.90:6789/0 conn(0x7f63d80a98f0 legacy=0x7f63d81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:27.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 shutdown_connections 2024-03-25T19:22:27.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 >> 172.21.15.60:0/2546160643 conn(0x7f63d80b7ea0 msgr2=0x7f63d814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:27.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 shutdown_connections 2024-03-25T19:22:27.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 1 -- 172.21.15.60:0/2546160643 wait complete. 2024-03-25T19:22:27.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:27.063+0000 7f63dcc4e640 1 librados: shutdown 2024-03-25T19:22:27.073 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:33.880 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.994+0000 7fc96e993640 10 monclient: get_monmap_and_config 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.994+0000 7fc96e993640 10 monclient: build_initial_monmap 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.994+0000 7fc96e993640 10 monclient: monmap: 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:33.995315+0000 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:33.995315+0000 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:33.994 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 1 Processor -- start 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 1 -- start start 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 10 monclient: init 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 10 monclient(hunting): picked mon.noname-b con 0x7fc9680b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 10 monclient(hunting): picked mon.noname-a con 0x7fc9680b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.995+0000 7fc96e993640 10 monclient(hunting): picked mon.noname-c con 0x7fc96814be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.996+0000 7fc96e993640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680af940 con 0x7fc9680b1730 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.996+0000 7fc96e993640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680b0a70 con 0x7fc96814be90 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.996+0000 7fc96e993640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9680b0d50 con 0x7fc9680b4d60 2024-03-25T19:22:33.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.996+0000 7fc96e993640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:33.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.996+0000 7fc96e993640 10 monclient(hunting): authenticate will time out at 6481.094028s 2024-03-25T19:22:33.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.996+0000 7fc96cf30640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc96814be90 0x7fc96814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:35740/0 (socket says 172.21.15.60:35740) 2024-03-25T19:22:33.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:33.996+0000 7fc96cf30640 1 -- 172.21.15.60:0/3896280417 learned_addr learned my addr 172.21.15.60:0/3896280417 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:34.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.014+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78771171 0 0) 0x7fc9680af940 con 0x7fc9680b1730 2024-03-25T19:22:34.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.014+0000 7fc966ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.014+0000 7fc966ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.014+0000 7fc966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.014+0000 7fc966ffd640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:22:34.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.014+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950003520 con 0x7fc9680b1730 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2206956839 0 0) 0x7fc950003520 con 0x7fc9680b1730 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 >> v1:172.21.15.60:6790/0 conn(0x7fc9680b4d60 legacy=0x7fc96813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 >> v1:172.21.15.90:6789/0 conn(0x7fc96814be90 legacy=0x7fc96814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 1 monclient: found mon.noname-a 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:22:34.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9680b1030 con 0x7fc9680b1730 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc95c001690 con 0x7fc9680b1730 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc966ffd640 10 monclient: dump: 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:34.044 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:34.045 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc96e993640 5 monclient: authenticate success, global_id 4326 2024-03-25T19:22:34.045 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.043+0000 7fc96e993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc95c0017f0 con 0x7fc9680b1730 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc966ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc966ffd640 1 -- 172.21.15.60:0/3896280417 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc95c001c00 con 0x7fc9680b1730 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc96e993640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc96e993640 4 set_mon_vals no callback set 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc966ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.070+0000 7fc966ffd640 10 monclient: dump: 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.070 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.071 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.071 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:34.071 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:34.071 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:34.071 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:34.071 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:34.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.071+0000 7fc96e993640 10 monclient: shutdown 2024-03-25T19:22:34.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.071+0000 7fc96e993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:34.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.071+0000 7fc96e993640 1 -- 172.21.15.60:0/3896280417 >> v1:172.21.15.60:6789/0 conn(0x7fc9680b1730 legacy=0x7fc9680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.071+0000 7fc96e993640 1 -- 172.21.15.60:0/3896280417 shutdown_connections 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.071+0000 7fc96e993640 1 -- 172.21.15.60:0/3896280417 >> 172.21.15.60:0/3896280417 conn(0x7fc9680617a0 msgr2=0x7fc968061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.071+0000 7fc96e993640 1 -- 172.21.15.60:0/3896280417 shutdown_connections 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.071+0000 7fc96e993640 1 -- 172.21.15.60:0/3896280417 wait complete. 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.072+0000 7fc96e993640 10 monclient: build_initial_monmap 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.072+0000 7fc96e993640 10 monclient: monmap: 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:34.074 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.072+0000 7fc96e993640 1 librados: starting msgr at 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.072+0000 7fc96e993640 1 librados: starting objecter 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.072+0000 7fc96e993640 1 Processor -- start 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 1 -- start start 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 1 librados: setting wanted keys 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 1 librados: calling monclient init 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient: init 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient(hunting): picked mon.noname-c con 0x7fc9680b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient(hunting): picked mon.noname-a con 0x7fc9680b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:34.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient(hunting): picked mon.noname-b con 0x7fc96814be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9681420d0 con 0x7fc9680b4d60 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc968138270 con 0x7fc96814be90 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc968138400 con 0x7fc9680b1730 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc96e993640 10 monclient(hunting): authenticate will time out at 6481.171455s 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc967fff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fc9680b1730 0x7fc9681db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:53338/0 (socket says 172.21.15.60:53338) 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.073+0000 7fc967fff640 1 -- 172.21.15.60:0/1251268833 learned_addr learned my addr 172.21.15.60:0/1251268833 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.074+0000 7fc964ff9640 10 client.?.objecter ms_handle_connect 0x7fc9680b1730 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.074+0000 7fc964ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.074+0000 7fc964ff9640 10 client.?.objecter ms_handle_connect 0x7fc96814be90 2024-03-25T19:22:34.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.074+0000 7fc964ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:34.121 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:34.832 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.127 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:34.838 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.141+0000 7fc964ff9640 10 client.?.objecter ms_handle_connect 0x7fc9680b4d60 2024-03-25T19:22:34.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.141+0000 7fc964ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:34.166 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:34.877 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.175 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:34.829 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.181+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279179903 0 0) 0x7fc968138270 con 0x7fc96814be90 2024-03-25T19:22:34.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.181+0000 7fc964ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.181+0000 7fc964ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.181+0000 7fc964ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.181+0000 7fc964ff9640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:22:34.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.181+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc954004eb0 con 0x7fc96814be90 2024-03-25T19:22:34.183 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:34.837 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.186+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232717184 0 0) 0x7fc9681420d0 con 0x7fc9680b4d60 2024-03-25T19:22:34.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.186+0000 7fc964ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.186+0000 7fc964ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.186+0000 7fc964ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.186+0000 7fc964ff9640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:22:34.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.186+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc968138270 con 0x7fc9680b4d60 2024-03-25T19:22:34.190 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:34.901 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.193 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:34.905 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.198+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220413789 0 0) 0x7fc954004eb0 con 0x7fc96814be90 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.198+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9681420d0 con 0x7fc96814be90 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.198+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc960001b00 con 0x7fc96814be90 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.198+0000 7fc964ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.198+0000 7fc964ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.198+0000 7fc964ff9640 10 monclient(hunting): dump: 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:34.198 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:34.199 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:34.199 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:34.199 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:34.199 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.198+0000 7fc964ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:34.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.214+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264717209 0 0) 0x7fc968138400 con 0x7fc9680b1730 2024-03-25T19:22:34.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.214+0000 7fc964ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:22:34.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.214+0000 7fc964ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:34.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.214+0000 7fc964ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:34.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.214+0000 7fc964ff9640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:22:34.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.214+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc954004eb0 con 0x7fc9680b1730 2024-03-25T19:22:34.215 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:34.869 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.217 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:34.929 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 646718642 0 0) 0x7fc9681420d0 con 0x7fc96814be90 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 >> v1:172.21.15.60:6790/0 conn(0x7fc9680b1730 legacy=0x7fc9681db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 >> v1:172.21.15.60:6789/0 conn(0x7fc9680b4d60 legacy=0x7fc9681405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 1 monclient: found mon.b 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc968142480 con 0x7fc96814be90 2024-03-25T19:22:34.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 10 monclient: _finish_auth 0 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc964ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 10 monclient: _renew_subs 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9681426d0 con 0x7fc96814be90 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 10 monclient: _renew_subs 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9681dbae0 con 0x7fc96814be90 2024-03-25T19:22:34.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.226+0000 7fc96e993640 1 librados: init done 2024-03-25T19:22:34.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.227+0000 7fc96e993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:34.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.227+0000 7fc96e993640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:34.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.227+0000 7fc96e993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:34.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.227+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9681401f0 con 0x7fc96814be90 2024-03-25T19:22:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.513+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc960001f70 con 0x7fc96814be90 2024-03-25T19:22:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.513+0000 7fc964ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.513+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc960004d00 con 0x7fc96814be90 2024-03-25T19:22:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.513+0000 7fc964ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.513+0000 7fc964ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.513+0000 7fc964ff9640 10 monclient: dump: 2024-03-25T19:22:34.514 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.513+0000 7fc9667fc640 4 set_mon_vals no callback set 2024-03-25T19:22:34.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.515+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fc960004f20 con 0x7fc96814be90 2024-03-25T19:22:34.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.515+0000 7fc967fff640 10 monclient: get_auth_request con 0x7fc95406de00 auth_method 0 2024-03-25T19:22:34.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.515+0000 7fc964ff9640 10 client.4315.objecter ms_handle_connect 0x7fc95406de00 2024-03-25T19:22:34.519 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.520+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fc960054940 con 0x7fc96814be90 2024-03-25T19:22:34.519 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.520+0000 7fc964ff9640 10 client.4315.objecter ms_dispatch 0x7fc968000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:34.519 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.520+0000 7fc964ff9640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:34.519 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.520+0000 7fc964ff9640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:34.519 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.520+0000 7fc964ff9640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T19:22:34.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.521+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc960050450 con 0x7fc96814be90 2024-03-25T19:22:34.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.521+0000 7fc964ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:34.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.521+0000 7fc964ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:34.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.775+0000 7fc96e993640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:34.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.775+0000 7fc96e993640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:34.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.775+0000 7fc96e993640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:34.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.775+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9680627b0 con 0x7fc96814be90 2024-03-25T19:22:34.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.798+0000 7fc964ff9640 1 -- 172.21.15.60:0/1251268833 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fc960052380 con 0x7fc96814be90 2024-03-25T19:22:34.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.798+0000 7fc964ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:34.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.798+0000 7fc964ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:34.798 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:34.798 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.800+0000 7fc96e993640 10 librados: watch_flush enter 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.800+0000 7fc96e993640 10 librados: watch_flush exit 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.800+0000 7fc96e993640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.800+0000 7fc96e993640 10 client.4315.objecter successfully canceled tick 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 >> v1:172.21.15.90:6800/1827638754 conn(0x7fc95406de00 legacy=0x7fc9540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 10 monclient: shutdown 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 >> v1:172.21.15.90:6789/0 conn(0x7fc96814be90 legacy=0x7fc968134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 shutdown_connections 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 >> 172.21.15.60:0/1251268833 conn(0x7fc9680617a0 msgr2=0x7fc9680b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 shutdown_connections 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 1 -- 172.21.15.60:0/1251268833 wait complete. 2024-03-25T19:22:34.801 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:34.801+0000 7fc96e993640 1 librados: shutdown 2024-03-25T19:22:34.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:41.618 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.732+0000 7fbf0fda4640 10 monclient: get_monmap_and_config 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.732+0000 7fbf0fda4640 10 monclient: build_initial_monmap 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.732+0000 7fbf0fda4640 10 monclient: monmap: 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:41.733281+0000 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:41.733281+0000 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:41.732 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:41.733 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.733+0000 7fbf0fda4640 1 Processor -- start 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.733+0000 7fbf0fda4640 1 -- start start 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.733+0000 7fbf0fda4640 10 monclient: init 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.733+0000 7fbf0fda4640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.733+0000 7fbf0fda4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.733+0000 7fbf0fda4640 10 monclient(hunting): picked mon.noname-a con 0x7fbf080a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.734+0000 7fbf0fda4640 10 monclient(hunting): picked mon.noname-b con 0x7fbf0813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.734+0000 7fbf0fda4640 10 monclient(hunting): picked mon.noname-c con 0x7fbf081430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.734+0000 7fbf0fda4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf08142520 con 0x7fbf080a98f0 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.734+0000 7fbf0fda4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf080b28a0 con 0x7fbf081430c0 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.734+0000 7fbf0fda4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf080b2a30 con 0x7fbf0813f710 2024-03-25T19:22:41.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.734+0000 7fbf0fda4640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:41.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.734+0000 7fbf0fda4640 10 monclient(hunting): authenticate will time out at 6488.831957s 2024-03-25T19:22:41.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.737+0000 7fbf0e341640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbf081430c0 0x7fbf0814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:34626/0 (socket says 172.21.15.60:34626) 2024-03-25T19:22:41.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.737+0000 7fbf0e341640 1 -- 172.21.15.60:0/1097809506 learned_addr learned my addr 172.21.15.60:0/1097809506 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:41.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.821+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2093704413 0 0) 0x7fbf08142520 con 0x7fbf080a98f0 2024-03-25T19:22:41.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.821+0000 7fbf0cb3e640 10 monclient(hunting): _negotiate 2024-03-25T19:22:41.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.821+0000 7fbf0cb3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:41.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.821+0000 7fbf0cb3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:41.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.821+0000 7fbf0cb3e640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:22:41.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.821+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef4003520 con 0x7fbf080a98f0 2024-03-25T19:22:41.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.826+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804474123 0 0) 0x7fbf080b2a30 con 0x7fbf0813f710 2024-03-25T19:22:41.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.826+0000 7fbf0cb3e640 10 monclient(hunting): _negotiate 2024-03-25T19:22:41.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.826+0000 7fbf0cb3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:41.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.826+0000 7fbf0cb3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:41.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.826+0000 7fbf0cb3e640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:22:41.825 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.826+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf08142520 con 0x7fbf0813f710 2024-03-25T19:22:41.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.827+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284017976 0 0) 0x7fbf080b28a0 con 0x7fbf081430c0 2024-03-25T19:22:41.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.827+0000 7fbf0cb3e640 10 monclient(hunting): _negotiate 2024-03-25T19:22:41.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.827+0000 7fbf0cb3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:41.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.827+0000 7fbf0cb3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:41.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.827+0000 7fbf0cb3e640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:22:41.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.827+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf080b2a30 con 0x7fbf081430c0 2024-03-25T19:22:41.829 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939520704 0 0) 0x7fbef4003520 con 0x7fbf080a98f0 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 >> v1:172.21.15.60:6790/0 conn(0x7fbf0813f710 legacy=0x7fbf081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 >> v1:172.21.15.90:6789/0 conn(0x7fbf081430c0 legacy=0x7fbf0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 10 monclient: _finish_hunting 0 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 1 monclient: found mon.noname-a 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf080b44c0 con 0x7fbf080a98f0 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 10 monclient: _finish_auth 0 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 10 monclient: _check_auth_tickets 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbf04001690 con 0x7fbf080a98f0 2024-03-25T19:22:41.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0fda4640 5 monclient: authenticate success, global_id 4332 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0fda4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0cb3e640 10 monclient: dump: 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:41.831 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:41.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.830+0000 7fbf0fda4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:41.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf040017f0 con 0x7fbf080a98f0 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0cb3e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0cb3e640 1 -- 172.21.15.60:0/1097809506 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbf04001c00 con 0x7fbf080a98f0 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0cb3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0cb3e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0cb3e640 10 monclient: dump: 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:41.839 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:41.840 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:41.840 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:41.840 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0fda4640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:41.840 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.839+0000 7fbf0fda4640 4 set_mon_vals no callback set 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.840+0000 7fbf0fda4640 10 monclient: shutdown 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.840+0000 7fbf0fda4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.840+0000 7fbf0fda4640 1 -- 172.21.15.60:0/1097809506 >> v1:172.21.15.60:6789/0 conn(0x7fbf080a98f0 legacy=0x7fbf080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.840+0000 7fbf0fda4640 1 -- 172.21.15.60:0/1097809506 shutdown_connections 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.840+0000 7fbf0fda4640 1 -- 172.21.15.60:0/1097809506 >> 172.21.15.60:0/1097809506 conn(0x7fbf080b7ea0 msgr2=0x7fbf080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.840+0000 7fbf0fda4640 1 -- 172.21.15.60:0/1097809506 shutdown_connections 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.840+0000 7fbf0fda4640 1 -- 172.21.15.60:0/1097809506 wait complete. 2024-03-25T19:22:41.841 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 10 monclient: build_initial_monmap 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 10 monclient: monmap: 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 1 librados: starting msgr at 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 1 librados: starting objecter 2024-03-25T19:22:41.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 1 Processor -- start 2024-03-25T19:22:41.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 1 -- start start 2024-03-25T19:22:41.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 1 librados: setting wanted keys 2024-03-25T19:22:41.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 1 librados: calling monclient init 2024-03-25T19:22:41.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.841+0000 7fbf0fda4640 10 monclient: init 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 10 monclient(hunting): picked mon.noname-a con 0x7fbf080a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 10 monclient(hunting): picked mon.noname-c con 0x7fbf0813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 10 monclient(hunting): picked mon.noname-b con 0x7fbf081430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf081369b0 con 0x7fbf080a98f0 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf081366b0 con 0x7fbf081430c0 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf08133ad0 con 0x7fbf0813f710 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:41.844 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.842+0000 7fbf0fda4640 10 monclient(hunting): authenticate will time out at 6488.940576s 2024-03-25T19:22:41.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.850+0000 7fbf0e341640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbf081430c0 0x7fbf081333c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:34634/0 (socket says 172.21.15.60:34634) 2024-03-25T19:22:41.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.850+0000 7fbf0e341640 1 -- 172.21.15.60:0/4104755549 learned_addr learned my addr 172.21.15.60:0/4104755549 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:41.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.852+0000 7fbefe7fc640 10 client.?.objecter ms_handle_connect 0x7fbf081430c0 2024-03-25T19:22:41.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.852+0000 7fbefe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:41.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.855+0000 7fbefe7fc640 10 client.?.objecter ms_handle_connect 0x7fbf080a98f0 2024-03-25T19:22:41.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.855+0000 7fbefe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:41.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.856+0000 7fbefe7fc640 10 client.?.objecter ms_handle_connect 0x7fbf0813f710 2024-03-25T19:22:41.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.856+0000 7fbefe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:41.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.860+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521467931 0 0) 0x7fbf081366b0 con 0x7fbf081430c0 2024-03-25T19:22:41.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.860+0000 7fbefe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:41.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.860+0000 7fbefe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:41.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.860+0000 7fbefe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:41.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.860+0000 7fbefe7fc640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:22:41.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.860+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec004eb0 con 0x7fbf081430c0 2024-03-25T19:22:41.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.861+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421630552 0 0) 0x7fbf081369b0 con 0x7fbf080a98f0 2024-03-25T19:22:41.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.861+0000 7fbefe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:41.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.861+0000 7fbefe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:41.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.861+0000 7fbefe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:41.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.861+0000 7fbefe7fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:22:41.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.861+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf081366b0 con 0x7fbf080a98f0 2024-03-25T19:22:41.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.865+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308066654 0 0) 0x7fbf08133ad0 con 0x7fbf0813f710 2024-03-25T19:22:41.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.865+0000 7fbefe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:41.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.865+0000 7fbefe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:41.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.865+0000 7fbefe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:41.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.865+0000 7fbefe7fc640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:22:41.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.865+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf081369b0 con 0x7fbf0813f710 2024-03-25T19:22:41.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.872+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2969309126 0 0) 0x7fbeec004eb0 con 0x7fbf081430c0 2024-03-25T19:22:41.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.872+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf08133ad0 con 0x7fbf081430c0 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.872+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbf00001bd0 con 0x7fbf081430c0 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.872+0000 7fbefe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.872+0000 7fbefe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.872+0000 7fbefe7fc640 10 monclient(hunting): dump: 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:41.872 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:41.873 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:41.873 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:41.873 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.872+0000 7fbefe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:41.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.880+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1960730286 0 0) 0x7fbf08133ad0 con 0x7fbf081430c0 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 >> v1:172.21.15.60:6790/0 conn(0x7fbf0813f710 legacy=0x7fbf08132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 >> v1:172.21.15.60:6789/0 conn(0x7fbf080a98f0 legacy=0x7fbf081371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 1 monclient: found mon.b 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08136e40 con 0x7fbf081430c0 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbefe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 10 monclient: _renew_subs 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:41.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf081378f0 con 0x7fbf081430c0 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 10 monclient: _renew_subs 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf080b1480 con 0x7fbf081430c0 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.881+0000 7fbf0fda4640 1 librados: init done 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.882+0000 7fbf0fda4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.882+0000 7fbf0fda4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:41.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.882+0000 7fbf0fda4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:41.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:41.882+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf08137030 con 0x7fbf081430c0 2024-03-25T19:22:42.158 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf00002080 con 0x7fbf081430c0 2024-03-25T19:22:42.158 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbf00004d00 con 0x7fbf081430c0 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefe7fc640 10 monclient: dump: 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:42.159 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefffff640 4 set_mon_vals no callback set 2024-03-25T19:22:42.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.159+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fbf00004f20 con 0x7fbf081430c0 2024-03-25T19:22:42.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.160+0000 7fbf0db40640 10 monclient: get_auth_request con 0x7fbeec06dd90 auth_method 0 2024-03-25T19:22:42.160 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.160+0000 7fbefe7fc640 10 client.4321.objecter ms_handle_connect 0x7fbeec06dd90 2024-03-25T19:22:42.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fbf00054b60 con 0x7fbf081430c0 2024-03-25T19:22:42.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 10 client.4321.objecter ms_dispatch 0x7fbf08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:42.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:42.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:42.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:22:42.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf000504a0 con 0x7fbf081430c0 2024-03-25T19:22:42.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:42.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.165+0000 7fbefe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:42.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.421+0000 7fbf0fda4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:42.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.421+0000 7fbf0fda4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:42.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.421+0000 7fbf0fda4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:42.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.421+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf08136b60 con 0x7fbf081430c0 2024-03-25T19:22:42.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.442+0000 7fbefe7fc640 1 -- 172.21.15.60:0/4104755549 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fbf000525a0 con 0x7fbf081430c0 2024-03-25T19:22:42.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.442+0000 7fbefe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:42.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.442+0000 7fbefe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:42.442 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:42.442 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 10 librados: watch_flush enter 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 10 librados: watch_flush exit 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 >> v1:172.21.15.90:6800/1827638754 conn(0x7fbeec06dd90 legacy=0x7fbeec070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 10 monclient: shutdown 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 >> v1:172.21.15.90:6789/0 conn(0x7fbf081430c0 legacy=0x7fbf081333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:42.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 shutdown_connections 2024-03-25T19:22:42.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 >> 172.21.15.60:0/4104755549 conn(0x7fbf080b7ea0 msgr2=0x7fbf08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:42.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 shutdown_connections 2024-03-25T19:22:42.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 1 -- 172.21.15.60:0/4104755549 wait complete. 2024-03-25T19:22:42.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:42.445+0000 7fbf0fda4640 1 librados: shutdown 2024-03-25T19:22:42.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:49.264 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.382+0000 7ff448c4a640 10 monclient: get_monmap_and_config 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.382+0000 7ff448c4a640 10 monclient: build_initial_monmap 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.382+0000 7ff448c4a640 10 monclient: monmap: 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:49.383179+0000 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:49.383179+0000 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:49.382 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:49.385 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:49.385 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:49.385 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:49.386 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.383+0000 7ff448c4a640 1 Processor -- start 2024-03-25T19:22:49.386 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.383+0000 7ff448c4a640 1 -- start start 2024-03-25T19:22:49.386 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.383+0000 7ff448c4a640 10 monclient: init 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.383+0000 7ff448c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.383+0000 7ff448c4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff4440b1ef0 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff4441473a0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff444150070 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4440b2570 con 0x7ff4440b1ef0 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff444146c20 con 0x7ff444150070 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff444146e80 con 0x7ff4441473a0 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:49.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.384+0000 7ff448c4a640 10 monclient(hunting): authenticate will time out at 6496.481976s 2024-03-25T19:22:49.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.389+0000 7ff442d9d640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff444150070 0x7ff444152460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:50156/0 (socket says 172.21.15.60:50156) 2024-03-25T19:22:49.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.389+0000 7ff442d9d640 1 -- 172.21.15.60:0/1549837583 learned_addr learned my addr 172.21.15.60:0/1549837583 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:49.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.398+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961436307 0 0) 0x7ff444146c20 con 0x7ff444150070 2024-03-25T19:22:49.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.398+0000 7ff44159a640 10 monclient(hunting): _negotiate 2024-03-25T19:22:49.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.398+0000 7ff44159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:49.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.398+0000 7ff44159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:49.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.398+0000 7ff44159a640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:22:49.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.398+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42c003520 con 0x7ff444150070 2024-03-25T19:22:49.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 985866162 0 0) 0x7ff42c003520 con 0x7ff444150070 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 >> v1:172.21.15.60:6790/0 conn(0x7ff4441473a0 legacy=0x7ff444143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 >> v1:172.21.15.60:6789/0 conn(0x7ff4440b1ef0 legacy=0x7ff4440b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 10 monclient: _finish_hunting 0 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 1 monclient: found mon.noname-c 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44406e1b0 con 0x7ff444150070 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 10 monclient: _finish_auth 0 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 10 monclient: _check_auth_tickets 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff43c001a10 con 0x7ff444150070 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff448c4a640 5 monclient: authenticate success, global_id 4324 2024-03-25T19:22:49.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff448c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff44159a640 10 monclient: dump: 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:49.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.410+0000 7ff448c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:49.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.419+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff43c001be0 con 0x7ff444150070 2024-03-25T19:22:49.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.419+0000 7ff44159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:49.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.419+0000 7ff44159a640 1 -- 172.21.15.60:0/1549837583 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff43c002030 con 0x7ff444150070 2024-03-25T19:22:49.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.419+0000 7ff44159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:49.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.419+0000 7ff44159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:49.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.419+0000 7ff44159a640 10 monclient: dump: 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.420+0000 7ff448c4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:49.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.420+0000 7ff448c4a640 4 set_mon_vals no callback set 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.420+0000 7ff448c4a640 10 monclient: shutdown 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.420+0000 7ff448c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.420+0000 7ff448c4a640 1 -- 172.21.15.60:0/1549837583 >> v1:172.21.15.90:6789/0 conn(0x7ff444150070 legacy=0x7ff444152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.420+0000 7ff448c4a640 1 -- 172.21.15.60:0/1549837583 shutdown_connections 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.421+0000 7ff448c4a640 1 -- 172.21.15.60:0/1549837583 >> 172.21.15.60:0/1549837583 conn(0x7ff4440b7ea0 msgr2=0x7ff4440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.421+0000 7ff448c4a640 1 -- 172.21.15.60:0/1549837583 shutdown_connections 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.421+0000 7ff448c4a640 1 -- 172.21.15.60:0/1549837583 wait complete. 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.421+0000 7ff448c4a640 10 monclient: build_initial_monmap 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.421+0000 7ff448c4a640 10 monclient: monmap: 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:49.423 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.422+0000 7ff448c4a640 1 librados: starting msgr at 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.422+0000 7ff448c4a640 1 librados: starting objecter 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.422+0000 7ff448c4a640 1 Processor -- start 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.422+0000 7ff448c4a640 1 -- start start 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.422+0000 7ff448c4a640 1 librados: setting wanted keys 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.422+0000 7ff448c4a640 1 librados: calling monclient init 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.422+0000 7ff448c4a640 10 monclient: init 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:49.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff4440b1ef0 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff4441473a0 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff444150070 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff444144560 con 0x7ff4441473a0 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44406ff20 con 0x7ff444150070 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff444144800 con 0x7ff4440b1ef0 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:49.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.423+0000 7ff448c4a640 10 monclient(hunting): authenticate will time out at 6496.521219s 2024-03-25T19:22:49.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.431+0000 7ff442d9d640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff444150070 0x7ff4441e7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:50208/0 (socket says 172.21.15.60:50208) 2024-03-25T19:22:49.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.431+0000 7ff442d9d640 1 -- 172.21.15.60:0/971175111 learned_addr learned my addr 172.21.15.60:0/971175111 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:49.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.432+0000 7ff432ffd640 10 client.?.objecter ms_handle_connect 0x7ff444150070 2024-03-25T19:22:49.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.432+0000 7ff432ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:49.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.437+0000 7ff432ffd640 10 client.?.objecter ms_handle_connect 0x7ff4440b1ef0 2024-03-25T19:22:49.436 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.437+0000 7ff432ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:49.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.438+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452847901 0 0) 0x7ff44406ff20 con 0x7ff444150070 2024-03-25T19:22:49.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.438+0000 7ff432ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:49.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.438+0000 7ff432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:49.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.438+0000 7ff432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:49.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.438+0000 7ff432ffd640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:22:49.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.438+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff424004eb0 con 0x7ff444150070 2024-03-25T19:22:49.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.440+0000 7ff432ffd640 10 client.?.objecter ms_handle_connect 0x7ff4441473a0 2024-03-25T19:22:49.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.440+0000 7ff432ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:49.444 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490975241 0 0) 0x7ff424004eb0 con 0x7ff444150070 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff44406ff20 con 0x7ff444150070 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff43c001ef0 con 0x7ff444150070 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): dump: 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:49.445 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:49.446 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:49.446 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:49.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:49.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027900123 0 0) 0x7ff444144800 con 0x7ff4440b1ef0 2024-03-25T19:22:49.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:49.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:49.447 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:49.447 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:22:49.447 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.445+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff424004eb0 con 0x7ff4440b1ef0 2024-03-25T19:22:49.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.448+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177390023 0 0) 0x7ff444144560 con 0x7ff4441473a0 2024-03-25T19:22:49.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.448+0000 7ff432ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:22:49.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.448+0000 7ff432ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:49.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.448+0000 7ff432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:49.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.448+0000 7ff432ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:22:49.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.448+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff444144800 con 0x7ff4441473a0 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3647020101 0 0) 0x7ff444144800 con 0x7ff4441473a0 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff444144560 con 0x7ff4441473a0 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff438001510 con 0x7ff4441473a0 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 10 monclient(hunting): dump: 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.478 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2733256176 0 0) 0x7ff44406ff20 con 0x7ff444150070 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 >> v1:172.21.15.60:6790/0 conn(0x7ff4440b1ef0 legacy=0x7ff44406f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 >> v1:172.21.15.60:6789/0 conn(0x7ff4441473a0 legacy=0x7ff444138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 monclient: found mon.b 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4440716e0 con 0x7ff444150070 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 10 monclient: _finish_auth 0 2024-03-25T19:22:49.479 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 5 monclient: authenticate success, global_id 4327 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 10 monclient: _renew_subs 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff444146560 con 0x7ff444150070 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 10 monclient: _renew_subs 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44413f5e0 con 0x7ff444150070 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.477+0000 7ff448c4a640 1 librados: init done 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.478+0000 7ff448c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.478+0000 7ff448c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.478+0000 7ff448c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:49.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.478+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff444146240 con 0x7ff444150070 2024-03-25T19:22:49.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.760+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff43c004e10 con 0x7ff444150070 2024-03-25T19:22:49.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.760+0000 7ff432ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.760+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff43c005120 con 0x7ff444150070 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.760+0000 7ff432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.760+0000 7ff432ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.760+0000 7ff432ffd640 10 monclient: dump: 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:49.760 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:49.761 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:49.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.760+0000 7ff440d99640 4 set_mon_vals no callback set 2024-03-25T19:22:49.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.761+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7ff43c01b0f0 con 0x7ff444150070 2024-03-25T19:22:49.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.762+0000 7ff44259c640 10 monclient: get_auth_request con 0x7ff42406df30 auth_method 0 2024-03-25T19:22:49.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.762+0000 7ff432ffd640 10 client.4327.objecter ms_handle_connect 0x7ff42406df30 2024-03-25T19:22:49.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.763+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7ff43c054be0 con 0x7ff444150070 2024-03-25T19:22:49.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.763+0000 7ff432ffd640 10 client.4327.objecter ms_dispatch 0x7ff444000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:49.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.763+0000 7ff432ffd640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:49.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.763+0000 7ff432ffd640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:49.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.763+0000 7ff432ffd640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T19:22:49.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.767+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff43c0506f0 con 0x7ff444150070 2024-03-25T19:22:49.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.767+0000 7ff432ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:49.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.767+0000 7ff432ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:49.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.990+0000 7ff448c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:49.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.990+0000 7ff448c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:49.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.990+0000 7ff448c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:22:49.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:49.990+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff444071950 con 0x7ff444150070 2024-03-25T19:22:50.011 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.011+0000 7ff432ffd640 1 -- 172.21.15.60:0/971175111 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7ff43c052620 con 0x7ff444150070 2024-03-25T19:22:50.011 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.011+0000 7ff432ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:50.011 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.011+0000 7ff432ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:50.011 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:50.011 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:50.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 10 librados: watch_flush enter 2024-03-25T19:22:50.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 10 librados: watch_flush exit 2024-03-25T19:22:50.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T19:22:50.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 10 client.4327.objecter successfully canceled tick 2024-03-25T19:22:50.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 >> v1:172.21.15.90:6800/1827638754 conn(0x7ff42406df30 legacy=0x7ff4240703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:50.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 10 monclient: shutdown 2024-03-25T19:22:50.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:50.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 >> v1:172.21.15.90:6789/0 conn(0x7ff444150070 legacy=0x7ff4441e7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:50.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 shutdown_connections 2024-03-25T19:22:50.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 >> 172.21.15.60:0/971175111 conn(0x7ff4440b7ea0 msgr2=0x7ff4440b4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:50.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 shutdown_connections 2024-03-25T19:22:50.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 1 -- 172.21.15.60:0/971175111 wait complete. 2024-03-25T19:22:50.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:50.014+0000 7ff448c4a640 1 librados: shutdown 2024-03-25T19:22:50.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:22:54.102 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:54.813 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.108 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:54.820 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.148 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:02:54.859 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.164 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:54.818 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.170 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:54.824 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.171 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:54.883 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.178 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:54.889 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.196 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:02:54.850 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:54.204 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:02:54.915 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:22:56.832 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.947+0000 7fedbbcc0640 10 monclient: get_monmap_and_config 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.947+0000 7fedbbcc0640 10 monclient: build_initial_monmap 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.947+0000 7fedbbcc0640 10 monclient: monmap: 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:22:56.948306+0000 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:22:56.948306+0000 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:22:56.947 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:22:56.948 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.948+0000 7fedbbcc0640 1 Processor -- start 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.948+0000 7fedbbcc0640 1 -- start start 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.948+0000 7fedbbcc0640 10 monclient: init 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 10 monclient(hunting): picked mon.noname-a con 0x7fedb40b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 10 monclient(hunting): picked mon.noname-b con 0x7fedb40b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 10 monclient(hunting): picked mon.noname-c con 0x7fedb414be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb40af940 con 0x7fedb40b4d60 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb40b0a70 con 0x7fedb414be90 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb40b0d50 con 0x7fedb40b1730 2024-03-25T19:22:56.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:56.950 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.949+0000 7fedbbcc0640 10 monclient(hunting): authenticate will time out at 6504.047165s 2024-03-25T19:22:56.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.964+0000 7fedba25d640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fedb414be90 0x7fedb414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:42370/0 (socket says 172.21.15.60:42370) 2024-03-25T19:22:56.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.964+0000 7fedba25d640 1 -- 172.21.15.60:0/1622155920 learned_addr learned my addr 172.21.15.60:0/1622155920 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:56.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.981+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103070194 0 0) 0x7fedb40b0a70 con 0x7fedb414be90 2024-03-25T19:22:56.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.981+0000 7fedb8a5a640 10 monclient(hunting): _negotiate 2024-03-25T19:22:56.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.981+0000 7fedb8a5a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:56.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.981+0000 7fedb8a5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:56.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.981+0000 7fedb8a5a640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:22:56.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.981+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0003520 con 0x7fedb414be90 2024-03-25T19:22:56.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.989+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231114484 0 0) 0x7fedb40af940 con 0x7fedb40b4d60 2024-03-25T19:22:56.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): _negotiate 2024-03-25T19:22:56.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:56.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:56.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:22:56.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb40b0a70 con 0x7fedb40b4d60 2024-03-25T19:22:56.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313185092 0 0) 0x7fedb40b0d50 con 0x7fedb40b1730 2024-03-25T19:22:56.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): _negotiate 2024-03-25T19:22:56.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:56.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:56.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:22:56.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:56.990+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb40af940 con 0x7fedb40b1730 2024-03-25T19:22:57.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2867728142 0 0) 0x7feda0003520 con 0x7fedb414be90 2024-03-25T19:22:57.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 >> v1:172.21.15.60:6790/0 conn(0x7fedb40b1730 legacy=0x7fedb40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:57.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 >> v1:172.21.15.60:6789/0 conn(0x7fedb40b4d60 legacy=0x7fedb413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:57.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 10 monclient: _finish_hunting 0 2024-03-25T19:22:57.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 1 monclient: found mon.noname-c 2024-03-25T19:22:57.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:57.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb40b1030 con 0x7fedb414be90 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 10 monclient: _finish_auth 0 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 10 monclient: _check_auth_tickets 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fedac001a10 con 0x7fedb414be90 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedb8a5a640 10 monclient: dump: 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:57.001 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:57.002 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:57.002 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:57.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedbbcc0640 5 monclient: authenticate success, global_id 4330 2024-03-25T19:22:57.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.000+0000 7fedbbcc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedac001b70 con 0x7fedb414be90 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedb8a5a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedb8a5a640 1 -- 172.21.15.60:0/1622155920 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fedac001f80 con 0x7fedb414be90 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedb8a5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedb8a5a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedb8a5a640 10 monclient: dump: 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:57.029 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedbbcc0640 10 monclient: get_monmap_and_config success 2024-03-25T19:22:57.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.029+0000 7fedbbcc0640 4 set_mon_vals no callback set 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.030+0000 7fedbbcc0640 10 monclient: shutdown 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.030+0000 7fedbbcc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.030+0000 7fedbbcc0640 1 -- 172.21.15.60:0/1622155920 >> v1:172.21.15.90:6789/0 conn(0x7fedb414be90 legacy=0x7fedb414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.030+0000 7fedbbcc0640 1 -- 172.21.15.60:0/1622155920 shutdown_connections 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.030+0000 7fedbbcc0640 1 -- 172.21.15.60:0/1622155920 >> 172.21.15.60:0/1622155920 conn(0x7fedb40617a0 msgr2=0x7fedb4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.030+0000 7fedbbcc0640 1 -- 172.21.15.60:0/1622155920 shutdown_connections 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.030+0000 7fedbbcc0640 1 -- 172.21.15.60:0/1622155920 wait complete. 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 10 monclient: build_initial_monmap 2024-03-25T19:22:57.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 10 monclient: monmap: 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 1 librados: starting msgr at 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 1 librados: starting objecter 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 1 Processor -- start 2024-03-25T19:22:57.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 1 -- start start 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 1 librados: setting wanted keys 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 1 librados: calling monclient init 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.031+0000 7fedbbcc0640 10 monclient: init 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 10 monclient: _reopen_session rank -1 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 10 monclient(hunting): picked mon.noname-a con 0x7fedb40b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 10 monclient(hunting): picked mon.noname-c con 0x7fedb40b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 10 monclient(hunting): picked mon.noname-b con 0x7fedb414be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb4140c10 con 0x7fedb40b1730 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb4134900 con 0x7fedb414be90 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb4141a70 con 0x7fedb40b4d60 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 10 monclient(hunting): _renew_subs 2024-03-25T19:22:57.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.032+0000 7fedbbcc0640 10 monclient(hunting): authenticate will time out at 6504.130522s 2024-03-25T19:22:57.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.052+0000 7fedb9a5c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fedb40b1730 0x7fedb4142170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:55036/0 (socket says 172.21.15.60:55036) 2024-03-25T19:22:57.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.052+0000 7fedb9a5c640 1 -- 172.21.15.60:0/4195307654 learned_addr learned my addr 172.21.15.60:0/4195307654 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:22:57.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.055+0000 7fedaa7fc640 10 client.?.objecter ms_handle_connect 0x7fedb40b1730 2024-03-25T19:22:57.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.055+0000 7fedaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:57.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.059+0000 7fedaa7fc640 10 client.?.objecter ms_handle_connect 0x7fedb414be90 2024-03-25T19:22:57.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.059+0000 7fedaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:57.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.060+0000 7fedaa7fc640 10 client.?.objecter ms_handle_connect 0x7fedb40b4d60 2024-03-25T19:22:57.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.060+0000 7fedaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:22:57.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.067+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985112625 0 0) 0x7fedb4140c10 con 0x7fedb40b1730 2024-03-25T19:22:57.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.067+0000 7fedaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:57.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.067+0000 7fedaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:57.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.067+0000 7fedaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:57.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.067+0000 7fedaa7fc640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:22:57.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.067+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed98004eb0 con 0x7fedb40b1730 2024-03-25T19:22:57.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.068+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784479436 0 0) 0x7fedb4141a70 con 0x7fedb40b4d60 2024-03-25T19:22:57.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.068+0000 7fedaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:57.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.068+0000 7fedaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:57.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.068+0000 7fedaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:57.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.068+0000 7fedaa7fc640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:22:57.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.068+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb4140c10 con 0x7fedb40b4d60 2024-03-25T19:22:57.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.079+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771247186 0 0) 0x7fedb4134900 con 0x7fedb414be90 2024-03-25T19:22:57.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.079+0000 7fedaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:22:57.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.079+0000 7fedaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:22:57.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.079+0000 7fedaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:22:57.078 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.079+0000 7fedaa7fc640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:22:57.079 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.079+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedb4141a70 con 0x7fedb414be90 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.092+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3983447301 0 0) 0x7fedb4140c10 con 0x7fedb40b4d60 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.092+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedb4134900 con 0x7fedb40b4d60 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.092+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7feda4001870 con 0x7fedb40b4d60 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.092+0000 7fedaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.092+0000 7fedaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.092+0000 7fedaa7fc640 10 monclient(hunting): dump: 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.092 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:57.093 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:57.093 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:57.093 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:57.093 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:57.093 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:57.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.092+0000 7fedaa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:22:57.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.097+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359274111 0 0) 0x7fed98004eb0 con 0x7fedb40b1730 2024-03-25T19:22:57.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.097+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedb4140c10 con 0x7fedb40b1730 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.097+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fedb0001a30 con 0x7fedb40b1730 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.097+0000 7fedaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.097+0000 7fedaa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.097+0000 7fedaa7fc640 10 monclient(hunting): dump: 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:57.097 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.107+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3886351686 0 0) 0x7fedb4141a70 con 0x7fedb414be90 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.107+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed98004eb0 con 0x7fedb414be90 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.107+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fedac001e80 con 0x7fedb414be90 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.107+0000 7fedaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.107+0000 7fedaa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.107+0000 7fedaa7fc640 10 monclient(hunting): dump: 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:57.107 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:57.108 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:57.108 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:57.108 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:57.108 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:57.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.120+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3099811137 0 0) 0x7fedb4140c10 con 0x7fedb40b1730 2024-03-25T19:22:57.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 >> v1:172.21.15.60:6790/0 conn(0x7fedb40b4d60 legacy=0x7fedb4133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 >> v1:172.21.15.90:6789/0 conn(0x7fedb414be90 legacy=0x7fedb41341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 1 monclient: found mon.a 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb4141d50 con 0x7fedb40b1730 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 10 monclient: _finish_auth 0 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 10 monclient: _renew_subs 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb4140380 con 0x7fedb40b1730 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:22:57.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 10 monclient: _renew_subs 2024-03-25T19:22:57.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:57.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb41389e0 con 0x7fedb40b1730 2024-03-25T19:22:57.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.121+0000 7fedbbcc0640 1 librados: init done 2024-03-25T19:22:57.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.122+0000 7fedbbcc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:57.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.122+0000 7fedbbcc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:57.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.122+0000 7fedbbcc0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:57.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.122+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb40627b0 con 0x7fedb40b1730 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.428+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedb0001c50 con 0x7fedb40b1730 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.428+0000 7fedaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.428+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fedb0003a60 con 0x7fedb40b1730 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.428+0000 7fedaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.428+0000 7fedaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.428+0000 7fedaa7fc640 10 monclient: dump: 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:22:57.428 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:22:57.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.428+0000 7fedabfff640 4 set_mon_vals no callback set 2024-03-25T19:22:57.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.429+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fedb0003c40 con 0x7fedb40b1730 2024-03-25T19:22:57.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.430+0000 7fedb925b640 10 monclient: get_auth_request con 0x7fed9806e160 auth_method 0 2024-03-25T19:22:57.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.430+0000 7fedaa7fc640 10 client.4347.objecter ms_handle_connect 0x7fed9806e160 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fedb00548d0 con 0x7fedb40b1730 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 10 client.4347.objecter ms_dispatch 0x7fedb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fedb00503e0 con 0x7fedb40b1730 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:22:57.437 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.437+0000 7fedaa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:22:57.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.669+0000 7fedbbcc0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:57.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.669+0000 7fedbbcc0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:57.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.669+0000 7fedbbcc0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:22:57.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.669+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedb4140510 con 0x7fedb40b1730 2024-03-25T19:22:57.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.720+0000 7fedaa7fc640 1 -- 172.21.15.60:0/4195307654 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fedb0052310 con 0x7fedb40b1730 2024-03-25T19:22:57.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.720+0000 7fedaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:22:57.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.720+0000 7fedaa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:22:57.720 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:22:57.721 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 10 librados: watch_flush enter 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 10 librados: watch_flush exit 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 10 client.4347.objecter successfully canceled tick 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 >> v1:172.21.15.90:6800/1827638754 conn(0x7fed9806e160 legacy=0x7fed98070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 10 monclient: shutdown 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 >> v1:172.21.15.60:6789/0 conn(0x7fedb40b1730 legacy=0x7fedb4142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 shutdown_connections 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 >> 172.21.15.60:0/4195307654 conn(0x7fedb40617a0 msgr2=0x7fedb413b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:22:57.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 shutdown_connections 2024-03-25T19:22:57.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 1 -- 172.21.15.60:0/4195307654 wait complete. 2024-03-25T19:22:57.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:22:57.723+0000 7fedbbcc0640 1 librados: shutdown 2024-03-25T19:22:57.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:04.543 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.656+0000 7ff898486640 10 monclient: get_monmap_and_config 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.656+0000 7ff898486640 10 monclient: build_initial_monmap 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.656+0000 7ff898486640 10 monclient: monmap: 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:04.658036+0000 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:04.658036+0000 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:04.657 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:04.658 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:04.658 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:04.658 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.657+0000 7ff898486640 1 Processor -- start 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.657+0000 7ff898486640 1 -- start start 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.657+0000 7ff898486640 10 monclient: init 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 10 monclient(hunting): picked mon.noname-a con 0x7ff8900a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 10 monclient(hunting): picked mon.noname-c con 0x7ff89013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 10 monclient(hunting): picked mon.noname-b con 0x7ff8901430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:04.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff890142520 con 0x7ff8900a98f0 2024-03-25T19:23:04.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900b28a0 con 0x7ff89013f710 2024-03-25T19:23:04.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8900b2a30 con 0x7ff8901430c0 2024-03-25T19:23:04.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:04.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.658+0000 7ff898486640 10 monclient(hunting): authenticate will time out at 6511.757587s 2024-03-25T19:23:04.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.672+0000 7ff896a23640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7ff8901430c0 0x7ff89014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:35094/0 (socket says 172.21.15.60:35094) 2024-03-25T19:23:04.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.672+0000 7ff896a23640 1 -- 172.21.15.60:0/369095825 learned_addr learned my addr 172.21.15.60:0/369095825 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:04.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.695+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261169032 0 0) 0x7ff8900b2a30 con 0x7ff8901430c0 2024-03-25T19:23:04.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.695+0000 7ff895220640 10 monclient(hunting): _negotiate 2024-03-25T19:23:04.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.695+0000 7ff895220640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:04.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.695+0000 7ff895220640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:04.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.695+0000 7ff895220640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:23:04.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.695+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff87c003520 con 0x7ff8901430c0 2024-03-25T19:23:04.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.704+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841423674 0 0) 0x7ff890142520 con 0x7ff8900a98f0 2024-03-25T19:23:04.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.704+0000 7ff895220640 10 monclient(hunting): _negotiate 2024-03-25T19:23:04.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.704+0000 7ff895220640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:04.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.704+0000 7ff895220640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:04.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.704+0000 7ff895220640 10 monclient(hunting): my global_id is 4350 2024-03-25T19:23:04.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.704+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8900b2a30 con 0x7ff8900a98f0 2024-03-25T19:23:04.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.709+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139827984 0 0) 0x7ff8900b28a0 con 0x7ff89013f710 2024-03-25T19:23:04.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.709+0000 7ff895220640 10 monclient(hunting): _negotiate 2024-03-25T19:23:04.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.709+0000 7ff895220640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:04.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.709+0000 7ff895220640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:04.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.709+0000 7ff895220640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:23:04.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.709+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890142520 con 0x7ff89013f710 2024-03-25T19:23:04.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436584047 0 0) 0x7ff87c003520 con 0x7ff8901430c0 2024-03-25T19:23:04.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 >> v1:172.21.15.90:6789/0 conn(0x7ff89013f710 legacy=0x7ff8901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:04.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 >> v1:172.21.15.60:6789/0 conn(0x7ff8900a98f0 legacy=0x7ff8900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:04.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 10 monclient: _finish_hunting 0 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 1 monclient: found mon.noname-b 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8900b44c0 con 0x7ff8901430c0 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 10 monclient: _finish_auth 0 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 10 monclient: _check_auth_tickets 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.720+0000 7ff895220640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.721+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff888001a10 con 0x7ff8901430c0 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.721+0000 7ff895220640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.721+0000 7ff895220640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.721+0000 7ff895220640 10 monclient: dump: 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.722 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.721+0000 7ff898486640 5 monclient: authenticate success, global_id 4331 2024-03-25T19:23:04.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.721+0000 7ff898486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff888001b70 con 0x7ff8901430c0 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff895220640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff895220640 1 -- 172.21.15.60:0/369095825 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff888001f80 con 0x7ff8901430c0 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff898486640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff898486640 4 set_mon_vals no callback set 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff895220640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff895220640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.745+0000 7ff895220640 10 monclient: dump: 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:04.746 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.747 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.747 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:04.747 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:04.747 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:04.747 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:04.747 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:04.747 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:04.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.746+0000 7ff898486640 10 monclient: shutdown 2024-03-25T19:23:04.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.746+0000 7ff898486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:04.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.746+0000 7ff898486640 1 -- 172.21.15.60:0/369095825 >> v1:172.21.15.60:6790/0 conn(0x7ff8901430c0 legacy=0x7ff89014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:04.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.746+0000 7ff898486640 1 -- 172.21.15.60:0/369095825 shutdown_connections 2024-03-25T19:23:04.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.746+0000 7ff898486640 1 -- 172.21.15.60:0/369095825 >> 172.21.15.60:0/369095825 conn(0x7ff8900b7ea0 msgr2=0x7ff8900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:04.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.746+0000 7ff898486640 1 -- 172.21.15.60:0/369095825 shutdown_connections 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.746+0000 7ff898486640 1 -- 172.21.15.60:0/369095825 wait complete. 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.747+0000 7ff898486640 10 monclient: build_initial_monmap 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.747+0000 7ff898486640 10 monclient: monmap: 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.747+0000 7ff898486640 1 librados: starting msgr at 2024-03-25T19:23:04.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.747+0000 7ff898486640 1 librados: starting objecter 2024-03-25T19:23:04.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 1 Processor -- start 2024-03-25T19:23:04.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 1 -- start start 2024-03-25T19:23:04.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 1 librados: setting wanted keys 2024-03-25T19:23:04.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 1 librados: calling monclient init 2024-03-25T19:23:04.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 10 monclient: init 2024-03-25T19:23:04.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:04.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:04.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 10 monclient(hunting): picked mon.noname-c con 0x7ff8900a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:04.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.748+0000 7ff898486640 10 monclient(hunting): picked mon.noname-b con 0x7ff89013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:04.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.749+0000 7ff898486640 10 monclient(hunting): picked mon.noname-a con 0x7ff8901430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:04.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.749+0000 7ff898486640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8901396e0 con 0x7ff8901430c0 2024-03-25T19:23:04.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.749+0000 7ff898486640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff890138530 con 0x7ff89013f710 2024-03-25T19:23:04.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.749+0000 7ff898486640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8901399c0 con 0x7ff8900a98f0 2024-03-25T19:23:04.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.749+0000 7ff898486640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:04.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.749+0000 7ff898486640 10 monclient(hunting): authenticate will time out at 6511.848021s 2024-03-25T19:23:04.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.758+0000 7ff896a23640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff8901430c0 0x7ff8901e1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:60068/0 (socket says 172.21.15.60:60068) 2024-03-25T19:23:04.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.758+0000 7ff896a23640 1 -- 172.21.15.60:0/30840618 learned_addr learned my addr 172.21.15.60:0/30840618 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:04.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.763+0000 7ff886ffd640 10 client.?.objecter ms_handle_connect 0x7ff89013f710 2024-03-25T19:23:04.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.763+0000 7ff886ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:04.765 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.765+0000 7ff886ffd640 10 client.?.objecter ms_handle_connect 0x7ff8900a98f0 2024-03-25T19:23:04.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.765+0000 7ff886ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:04.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.766+0000 7ff886ffd640 10 client.?.objecter ms_handle_connect 0x7ff8901430c0 2024-03-25T19:23:04.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.766+0000 7ff886ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:04.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.772+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742992222 0 0) 0x7ff890138530 con 0x7ff89013f710 2024-03-25T19:23:04.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.772+0000 7ff886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:04.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.772+0000 7ff886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:04.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.772+0000 7ff886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:04.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.772+0000 7ff886ffd640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:23:04.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.772+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff874004eb0 con 0x7ff89013f710 2024-03-25T19:23:04.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.775+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 105495724 0 0) 0x7ff8901396e0 con 0x7ff8901430c0 2024-03-25T19:23:04.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.775+0000 7ff886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:04.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.775+0000 7ff886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:04.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.775+0000 7ff886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:04.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.775+0000 7ff886ffd640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:23:04.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.775+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890138530 con 0x7ff8901430c0 2024-03-25T19:23:04.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.779+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214554551 0 0) 0x7ff8901399c0 con 0x7ff8900a98f0 2024-03-25T19:23:04.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.779+0000 7ff886ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:04.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.779+0000 7ff886ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:04.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.779+0000 7ff886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:04.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.779+0000 7ff886ffd640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:23:04.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.779+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8901396e0 con 0x7ff8900a98f0 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.780+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3240388581 0 0) 0x7ff874004eb0 con 0x7ff89013f710 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.780+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8901399c0 con 0x7ff89013f710 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.780+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff8800018f0 con 0x7ff89013f710 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.780+0000 7ff886ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.780+0000 7ff886ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.780+0000 7ff886ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:04.781 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:04.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.780+0000 7ff886ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:04.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.793+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3171859663 0 0) 0x7ff890138530 con 0x7ff8901430c0 2024-03-25T19:23:04.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.793+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff874004eb0 con 0x7ff8901430c0 2024-03-25T19:23:04.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.793+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff888001e80 con 0x7ff8901430c0 2024-03-25T19:23:04.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.793+0000 7ff886ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:04.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.793+0000 7ff886ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.793+0000 7ff886ffd640 10 monclient(hunting): dump: 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:04.794 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:04.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.796+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1122766694 0 0) 0x7ff8901399c0 con 0x7ff89013f710 2024-03-25T19:23:04.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.796+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 >> v1:172.21.15.60:6790/0 conn(0x7ff8900a98f0 legacy=0x7ff890137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:04.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 >> v1:172.21.15.60:6789/0 conn(0x7ff8901430c0 legacy=0x7ff8901e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:04.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:04.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 1 monclient: found mon.b 2024-03-25T19:23:04.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:04.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890138a50 con 0x7ff89013f710 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 10 monclient: _renew_subs 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8901380a0 con 0x7ff89013f710 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 10 monclient: _renew_subs 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8901db140 con 0x7ff89013f710 2024-03-25T19:23:04.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.797+0000 7ff898486640 1 librados: init done 2024-03-25T19:23:04.799 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.798+0000 7ff898486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:04.799 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.798+0000 7ff898486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:04.799 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.798+0000 7ff898486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:04.799 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:04.798+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890138ca0 con 0x7ff89013f710 2024-03-25T19:23:05.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.091+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff880003610 con 0x7ff89013f710 2024-03-25T19:23:05.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.091+0000 7ff886ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:05.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.091+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff880003780 con 0x7ff89013f710 2024-03-25T19:23:05.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.091+0000 7ff886ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:05.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.091+0000 7ff886ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:05.092 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.091+0000 7ff886ffd640 10 monclient: dump: 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:05.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.091+0000 7ff894a1f640 4 set_mon_vals no callback set 2024-03-25T19:23:05.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.092+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7ff880019790 con 0x7ff89013f710 2024-03-25T19:23:05.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.093+0000 7ff896222640 10 monclient: get_auth_request con 0x7ff87406df70 auth_method 0 2024-03-25T19:23:05.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.093+0000 7ff886ffd640 10 client.4339.objecter ms_handle_connect 0x7ff87406df70 2024-03-25T19:23:05.098 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.097+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7ff8800534b0 con 0x7ff89013f710 2024-03-25T19:23:05.098 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.097+0000 7ff886ffd640 10 client.4339.objecter ms_dispatch 0x7ff890000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:05.098 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.097+0000 7ff886ffd640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:05.098 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.097+0000 7ff886ffd640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:05.098 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.097+0000 7ff886ffd640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-25T19:23:05.098 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.098+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff88004edf0 con 0x7ff89013f710 2024-03-25T19:23:05.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.098+0000 7ff886ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:05.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.098+0000 7ff886ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:05.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.352+0000 7ff898486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:05.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.352+0000 7ff898486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:05.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.352+0000 7ff898486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:05.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.352+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8900627b0 con 0x7ff89013f710 2024-03-25T19:23:05.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.375+0000 7ff886ffd640 1 -- 172.21.15.60:0/30840618 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7ff880050ef0 con 0x7ff89013f710 2024-03-25T19:23:05.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.376+0000 7ff886ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:05.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.376+0000 7ff886ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:05.376 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:05.377 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.378+0000 7ff898486640 10 librados: watch_flush enter 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 10 librados: watch_flush exit 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 10 client.4339.objecter successfully canceled tick 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 >> v1:172.21.15.90:6800/1827638754 conn(0x7ff87406df70 legacy=0x7ff874070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 10 monclient: shutdown 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 >> v1:172.21.15.90:6789/0 conn(0x7ff89013f710 legacy=0x7ff8901da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 shutdown_connections 2024-03-25T19:23:05.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 >> 172.21.15.60:0/30840618 conn(0x7ff8900b7ea0 msgr2=0x7ff890074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:05.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 shutdown_connections 2024-03-25T19:23:05.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 1 -- 172.21.15.60:0/30840618 wait complete. 2024-03-25T19:23:05.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:05.379+0000 7ff898486640 1 librados: shutdown 2024-03-25T19:23:05.392 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:12.199 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:12.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.309+0000 7f5779940640 10 monclient: get_monmap_and_config 2024-03-25T19:23:12.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.309+0000 7f5779940640 10 monclient: build_initial_monmap 2024-03-25T19:23:12.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.309+0000 7f5779940640 10 monclient: monmap: 2024-03-25T19:23:12.310 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:12.310 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:12.310 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:12.311583+0000 2024-03-25T19:23:12.310 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:12.311583+0000 2024-03-25T19:23:12.311 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:12.311 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:12.311 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:12.311 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:12.311 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:12.311 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:12.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.310+0000 7f5779940640 1 Processor -- start 2024-03-25T19:23:12.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.310+0000 7f5779940640 1 -- start start 2024-03-25T19:23:12.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.310+0000 7f5779940640 10 monclient: init 2024-03-25T19:23:12.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 10 monclient(hunting): picked mon.noname-a con 0x7f57740a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 10 monclient(hunting): picked mon.noname-b con 0x7f577413f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 10 monclient(hunting): picked mon.noname-c con 0x7f57741430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5774142520 con 0x7f57740a98f0 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57740b28a0 con 0x7f57741430c0 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57740b2a30 con 0x7f577413f710 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:12.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.311+0000 7f5779940640 10 monclient(hunting): authenticate will time out at 6519.410494s 2024-03-25T19:23:12.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.317+0000 7f576bfff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f577413f710 0x7f57741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:44406/0 (socket says 172.21.15.60:44406) 2024-03-25T19:23:12.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.317+0000 7f576bfff640 1 -- 172.21.15.60:0/1709631677 learned_addr learned my addr 172.21.15.60:0/1709631677 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:12.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.324+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200293190 0 0) 0x7f57740b2a30 con 0x7f577413f710 2024-03-25T19:23:12.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.324+0000 7f57727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:12.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.324+0000 7f57727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:12.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.324+0000 7f57727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:12.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.324+0000 7f57727fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:23:12.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.324+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5760003520 con 0x7f577413f710 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1489561221 0 0) 0x7f5760003520 con 0x7f577413f710 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 >> v1:172.21.15.90:6789/0 conn(0x7f57741430c0 legacy=0x7f577414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 >> v1:172.21.15.60:6789/0 conn(0x7f57740a98f0 legacy=0x7f57740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 1 monclient: found mon.noname-b 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57740b44c0 con 0x7f577413f710 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f575c001690 con 0x7f577413f710 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:12.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f57727fc640 10 monclient: dump: 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f5779940640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:23:12.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.331+0000 7f5779940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:12.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f575c001860 con 0x7f577413f710 2024-03-25T19:23:12.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f57727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:12.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f57727fc640 1 -- 172.21.15.60:0/1709631677 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f575c001cb0 con 0x7f577413f710 2024-03-25T19:23:12.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f5779940640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:12.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f5779940640 4 set_mon_vals no callback set 2024-03-25T19:23:12.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f57727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f57727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.335+0000 7f57727fc640 10 monclient: dump: 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:12.336 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:12.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.336+0000 7f5779940640 10 monclient: shutdown 2024-03-25T19:23:12.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.336+0000 7f5779940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.336+0000 7f5779940640 1 -- 172.21.15.60:0/1709631677 >> v1:172.21.15.60:6790/0 conn(0x7f577413f710 legacy=0x7f57741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.336+0000 7f5779940640 1 -- 172.21.15.60:0/1709631677 shutdown_connections 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.336+0000 7f5779940640 1 -- 172.21.15.60:0/1709631677 >> 172.21.15.60:0/1709631677 conn(0x7f57740b7ea0 msgr2=0x7f57740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.336+0000 7f5779940640 1 -- 172.21.15.60:0/1709631677 shutdown_connections 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.336+0000 7f5779940640 1 -- 172.21.15.60:0/1709631677 wait complete. 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 10 monclient: build_initial_monmap 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 10 monclient: monmap: 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:12.338 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 1 librados: starting msgr at 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 1 librados: starting objecter 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 1 Processor -- start 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 1 -- start start 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 1 librados: setting wanted keys 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 1 librados: calling monclient init 2024-03-25T19:23:12.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.337+0000 7f5779940640 10 monclient: init 2024-03-25T19:23:12.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:12.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:12.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 10 monclient(hunting): picked mon.noname-c con 0x7f57740a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:12.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 10 monclient(hunting): picked mon.noname-a con 0x7f577413f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:12.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 10 monclient(hunting): picked mon.noname-b con 0x7f57741430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:12.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57741d7630 con 0x7f577413f710 2024-03-25T19:23:12.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57741399c0 con 0x7f57741430c0 2024-03-25T19:23:12.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57741d41e0 con 0x7f57740a98f0 2024-03-25T19:23:12.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:12.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.338+0000 7f5779940640 10 monclient(hunting): authenticate will time out at 6519.437237s 2024-03-25T19:23:12.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.351+0000 7f5772ffd640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f57740a98f0 0x7f5774137e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:44426/0 (socket says 172.21.15.60:44426) 2024-03-25T19:23:12.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.351+0000 7f5772ffd640 1 -- 172.21.15.60:0/1766866729 learned_addr learned my addr 172.21.15.60:0/1766866729 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:12.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.351+0000 7f5770ff9640 10 client.?.objecter ms_handle_connect 0x7f57740a98f0 2024-03-25T19:23:12.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.351+0000 7f5770ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:12.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.356+0000 7f5770ff9640 10 client.?.objecter ms_handle_connect 0x7f57741430c0 2024-03-25T19:23:12.357 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.356+0000 7f5770ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:12.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.362+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660753122 0 0) 0x7f57741d41e0 con 0x7f57740a98f0 2024-03-25T19:23:12.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.362+0000 7f5770ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:12.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.362+0000 7f5770ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:12.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.362+0000 7f5770ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:12.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.362+0000 7f5770ff9640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:23:12.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.362+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5754004eb0 con 0x7f57740a98f0 2024-03-25T19:23:12.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.368+0000 7f5770ff9640 10 client.?.objecter ms_handle_connect 0x7f577413f710 2024-03-25T19:23:12.368 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.368+0000 7f5770ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:12.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.371+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356167479 0 0) 0x7f57741399c0 con 0x7f57741430c0 2024-03-25T19:23:12.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.371+0000 7f5770ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:12.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.371+0000 7f5770ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:12.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.371+0000 7f5770ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:12.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.371+0000 7f5770ff9640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:23:12.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.371+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57741d41e0 con 0x7f57741430c0 2024-03-25T19:23:12.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.387+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346553932 0 0) 0x7f5754004eb0 con 0x7f57740a98f0 2024-03-25T19:23:12.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.387+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57741399c0 con 0x7f57740a98f0 2024-03-25T19:23:12.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.387+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5764001960 con 0x7f57740a98f0 2024-03-25T19:23:12.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.387+0000 7f5770ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:12.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.387+0000 7f5770ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.387+0000 7f5770ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:12.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.387+0000 7f5770ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:12.391 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.391+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315444899 0 0) 0x7f57741d7630 con 0x7f577413f710 2024-03-25T19:23:12.392 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.391+0000 7f5770ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:12.392 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.391+0000 7f5770ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:12.392 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.391+0000 7f5770ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:12.392 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.391+0000 7f5770ff9640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:23:12.392 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.391+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5754004eb0 con 0x7f577413f710 2024-03-25T19:23:12.403 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.403+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3183796785 0 0) 0x7f57741d41e0 con 0x7f57741430c0 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.403+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57741d7630 con 0x7f57741430c0 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.403+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f576c001b10 con 0x7f57741430c0 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.403+0000 7f5770ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.403+0000 7f5770ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.403+0000 7f5770ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:12.404 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:12.405 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2819921414 0 0) 0x7f57741399c0 con 0x7f57740a98f0 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 >> v1:172.21.15.90:6789/0 conn(0x7f57741430c0 legacy=0x7f57741d47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 >> v1:172.21.15.60:6789/0 conn(0x7f577413f710 legacy=0x7f57741d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 1 monclient: found mon.c 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57741d4370 con 0x7f57740a98f0 2024-03-25T19:23:12.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5770ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 10 monclient: _renew_subs 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57741d8220 con 0x7f57740a98f0 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 10 monclient: _renew_subs 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57741d50c0 con 0x7f57740a98f0 2024-03-25T19:23:12.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.408+0000 7f5779940640 1 librados: init done 2024-03-25T19:23:12.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.409+0000 7f5779940640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:12.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.409+0000 7f5779940640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:12.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.409+0000 7f5779940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:12.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.409+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57740abfd0 con 0x7f57740a98f0 2024-03-25T19:23:12.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.720+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57640035f0 con 0x7f57740a98f0 2024-03-25T19:23:12.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.720+0000 7f5770ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.720+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f57640037a0 con 0x7f57740a98f0 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.720+0000 7f5770ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.720+0000 7f5770ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.720+0000 7f5770ff9640 10 monclient: dump: 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:12.721 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:12.722 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:12.722 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:12.722 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:12.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.720+0000 7f5771ffb640 4 set_mon_vals no callback set 2024-03-25T19:23:12.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.721+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f57640039c0 con 0x7f57740a98f0 2024-03-25T19:23:12.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.721+0000 7f576bfff640 10 monclient: get_auth_request con 0x7f575406dfb0 auth_method 0 2024-03-25T19:23:12.723 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.722+0000 7f5770ff9640 10 client.4340.objecter ms_handle_connect 0x7f575406dfb0 2024-03-25T19:23:12.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f57640545b0 con 0x7f57740a98f0 2024-03-25T19:23:12.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 10 client.4340.objecter ms_dispatch 0x7f5774000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:12.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:12.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:12.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T19:23:12.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57640500c0 con 0x7f57740a98f0 2024-03-25T19:23:12.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:12.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.727+0000 7f5770ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:12.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.976+0000 7f5779940640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:12.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.976+0000 7f5779940640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:12.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.976+0000 7f5779940640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:12.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:12.976+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57741d83b0 con 0x7f57740a98f0 2024-03-25T19:23:13.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.002+0000 7f5770ff9640 1 -- 172.21.15.60:0/1766866729 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f5764051ff0 con 0x7f57740a98f0 2024-03-25T19:23:13.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.002+0000 7f5770ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:13.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.002+0000 7f5770ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:13.003 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:13.003 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:13.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 10 librados: watch_flush enter 2024-03-25T19:23:13.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 10 librados: watch_flush exit 2024-03-25T19:23:13.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T19:23:13.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 10 client.4340.objecter successfully canceled tick 2024-03-25T19:23:13.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 >> v1:172.21.15.90:6800/1827638754 conn(0x7f575406dfb0 legacy=0x7f5754070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:13.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 10 monclient: shutdown 2024-03-25T19:23:13.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:13.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 >> v1:172.21.15.60:6790/0 conn(0x7f57740a98f0 legacy=0x7f5774137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:13.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 shutdown_connections 2024-03-25T19:23:13.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 >> 172.21.15.60:0/1766866729 conn(0x7f57740b7ea0 msgr2=0x7f5774074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:13.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 shutdown_connections 2024-03-25T19:23:13.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 1 -- 172.21.15.60:0/1766866729 wait complete. 2024-03-25T19:23:13.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:13.005+0000 7f5779940640 1 librados: shutdown 2024-03-25T19:23:13.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:14.193 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:14.904 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.204 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:14.915 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.226 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:14.938 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.250 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:14.904 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.256 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:14.967 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.265 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:14.919 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.267 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:14.978 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.293 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:14.947 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:14.301 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:15.013 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:19.826 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:19.941 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.941+0000 7f3e05c0f640 10 monclient: get_monmap_and_config 2024-03-25T19:23:19.941 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.941+0000 7f3e05c0f640 10 monclient: build_initial_monmap 2024-03-25T19:23:19.941 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.941+0000 7f3e05c0f640 10 monclient: monmap: 2024-03-25T19:23:19.941 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:19.941 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:19.942774+0000 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:19.942774+0000 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:19.942 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:19.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.942+0000 7f3e05c0f640 1 Processor -- start 2024-03-25T19:23:19.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.942+0000 7f3e05c0f640 1 -- start start 2024-03-25T19:23:19.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.942+0000 7f3e05c0f640 10 monclient: init 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f3e000a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f3e0013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f3e001430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e00142520 con 0x7f3e0013f710 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e000b28a0 con 0x7f3e001430c0 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e000b2a30 con 0x7f3e000a98f0 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:19.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.943+0000 7f3e05c0f640 10 monclient(hunting): authenticate will time out at 6527.042302s 2024-03-25T19:23:19.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.946+0000 7f3dff7fe640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f3e000a98f0 0x7f3e000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:33900/0 (socket says 172.21.15.60:33900) 2024-03-25T19:23:19.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.946+0000 7f3dff7fe640 1 -- 172.21.15.60:0/1517831087 learned_addr learned my addr 172.21.15.60:0/1517831087 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:19.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.956+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650192784 0 0) 0x7f3e000b2a30 con 0x7f3e000a98f0 2024-03-25T19:23:19.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.956+0000 7f3dfeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:19.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.956+0000 7f3dfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:19.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.956+0000 7f3dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:19.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.956+0000 7f3dfeffd640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:23:19.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.956+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dec003520 con 0x7f3e000a98f0 2024-03-25T19:23:19.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.961+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392966632 0 0) 0x7f3e000b28a0 con 0x7f3e001430c0 2024-03-25T19:23:19.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.961+0000 7f3dfeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:19.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.961+0000 7f3dfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:19.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.961+0000 7f3dfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:19.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.961+0000 7f3dfeffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:23:19.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.961+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e000b2a30 con 0x7f3e001430c0 2024-03-25T19:23:19.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 461536774 0 0) 0x7f3dec003520 con 0x7f3e000a98f0 2024-03-25T19:23:19.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 >> v1:172.21.15.90:6789/0 conn(0x7f3e001430c0 legacy=0x7f3e0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:19.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 >> v1:172.21.15.60:6789/0 conn(0x7f3e0013f710 legacy=0x7f3e001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 1 monclient: found mon.noname-b 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e000b44c0 con 0x7f3e000a98f0 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3df0001690 con 0x7f3e000a98f0 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3e05c0f640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:23:19.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3e05c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.964+0000 7f3dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.965+0000 7f3dfeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.965+0000 7f3dfeffd640 10 monclient: dump: 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:19.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.965+0000 7f3e05c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:19.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3df00017f0 con 0x7f3e000a98f0 2024-03-25T19:23:19.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3dfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:19.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3dfeffd640 1 -- 172.21.15.60:0/1517831087 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3df0001c00 con 0x7f3e000a98f0 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3e05c0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3e05c0f640 4 set_mon_vals no callback set 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3dfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3dfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.974+0000 7f3dfeffd640 10 monclient: dump: 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:19.976 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 10 monclient: shutdown 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 1 -- 172.21.15.60:0/1517831087 >> v1:172.21.15.60:6790/0 conn(0x7f3e000a98f0 legacy=0x7f3e000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 1 -- 172.21.15.60:0/1517831087 shutdown_connections 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 1 -- 172.21.15.60:0/1517831087 >> 172.21.15.60:0/1517831087 conn(0x7f3e000b7ea0 msgr2=0x7f3e000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 1 -- 172.21.15.60:0/1517831087 shutdown_connections 2024-03-25T19:23:19.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 1 -- 172.21.15.60:0/1517831087 wait complete. 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 10 monclient: build_initial_monmap 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.975+0000 7f3e05c0f640 10 monclient: monmap: 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:19.978 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.976+0000 7f3e05c0f640 1 librados: starting msgr at 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.976+0000 7f3e05c0f640 1 librados: starting objecter 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.976+0000 7f3e05c0f640 1 Processor -- start 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.976+0000 7f3e05c0f640 1 -- start start 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.976+0000 7f3e05c0f640 1 librados: setting wanted keys 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.976+0000 7f3e05c0f640 1 librados: calling monclient init 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.976+0000 7f3e05c0f640 10 monclient: init 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f3e000a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f3e0013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f3e001430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:19.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e001df080 con 0x7f3e0013f710 2024-03-25T19:23:19.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e00070420 con 0x7f3e000a98f0 2024-03-25T19:23:19.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0006f3c0 con 0x7f3e001430c0 2024-03-25T19:23:19.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:19.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.977+0000 7f3e05c0f640 10 monclient(hunting): authenticate will time out at 6527.076116s 2024-03-25T19:23:19.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.982+0000 7f3dfffff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f3e001430c0 0x7f3e001e64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:33922/0 (socket says 172.21.15.60:33922) 2024-03-25T19:23:19.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.982+0000 7f3dfffff640 1 -- 172.21.15.60:0/3714770075 learned_addr learned my addr 172.21.15.60:0/3714770075 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:19.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.983+0000 7f3dfcff9640 10 client.?.objecter ms_handle_connect 0x7f3e001430c0 2024-03-25T19:23:19.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.983+0000 7f3dfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:19.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.990+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730696786 0 0) 0x7f3e0006f3c0 con 0x7f3e001430c0 2024-03-25T19:23:19.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.990+0000 7f3dfcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:19.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.990+0000 7f3dfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:19.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.990+0000 7f3dfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:19.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.990+0000 7f3dfcff9640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:23:19.991 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.990+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de0004eb0 con 0x7f3e001430c0 2024-03-25T19:23:19.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.992+0000 7f3dfcff9640 10 client.?.objecter ms_handle_connect 0x7f3e000a98f0 2024-03-25T19:23:19.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.992+0000 7f3dfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:19.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.996+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 607310146 0 0) 0x7f3de0004eb0 con 0x7f3e001430c0 2024-03-25T19:23:19.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:19.996+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e0006f3c0 con 0x7f3e001430c0 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133207876 0 0) 0x7f3e00070420 con 0x7f3e000a98f0 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3de0004eb0 con 0x7f3e000a98f0 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 10 client.?.objecter ms_handle_connect 0x7f3e0013f710 2024-03-25T19:23:20.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.000+0000 7f3dfcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:20.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.002+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3df8001db0 con 0x7f3e001430c0 2024-03-25T19:23:20.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.002+0000 7f3dfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:20.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.002+0000 7f3dfcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:20.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.002+0000 7f3dfcff9640 10 monclient(hunting): dump: 2024-03-25T19:23:20.002 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:20.002 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:20.002 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:20.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.002+0000 7f3dfcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:20.011 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323268299 0 0) 0x7f3de0004eb0 con 0x7f3e000a98f0 2024-03-25T19:23:20.011 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e00070420 con 0x7f3e000a98f0 2024-03-25T19:23:20.011 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3df0001b00 con 0x7f3e000a98f0 2024-03-25T19:23:20.011 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 10 monclient(hunting): dump: 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:20.012 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1390970772 0 0) 0x7f3e0006f3c0 con 0x7f3e001430c0 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 >> v1:172.21.15.90:6789/0 conn(0x7f3e000a98f0 legacy=0x7f3e0006ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 >> v1:172.21.15.60:6789/0 conn(0x7f3e0013f710 legacy=0x7f3e001df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 monclient: found mon.c 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0006f550 con 0x7f3e001430c0 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3dfcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:20.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.011+0000 7f3e05c0f640 5 monclient: authenticate success, global_id 4346 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 10 monclient: _renew_subs 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0006e0a0 con 0x7f3e001430c0 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 10 monclient: _renew_subs 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e001dfe00 con 0x7f3e001430c0 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.012+0000 7f3e05c0f640 1 librados: init done 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.013+0000 7f3e05c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.013+0000 7f3e05c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.013+0000 7f3e05c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:20.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.013+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e001df230 con 0x7f3e001430c0 2024-03-25T19:23:20.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.330+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3df8001f10 con 0x7f3e001430c0 2024-03-25T19:23:20.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.330+0000 7f3dfcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:20.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.330+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3df8003cc0 con 0x7f3e001430c0 2024-03-25T19:23:20.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.330+0000 7f3dfcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:20.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.330+0000 7f3dfcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:20.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.330+0000 7f3dfcff9640 10 monclient: dump: 2024-03-25T19:23:20.331 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:20.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.330+0000 7f3dfe7fc640 4 set_mon_vals no callback set 2024-03-25T19:23:20.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.331+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f3df8003ee0 con 0x7f3e001430c0 2024-03-25T19:23:20.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.332+0000 7f3dff7fe640 10 monclient: get_auth_request con 0x7f3de006de60 auth_method 0 2024-03-25T19:23:20.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.332+0000 7f3dfcff9640 10 client.4346.objecter ms_handle_connect 0x7f3de006de60 2024-03-25T19:23:20.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.338+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f3df80549d0 con 0x7f3e001430c0 2024-03-25T19:23:20.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.338+0000 7f3dfcff9640 10 client.4346.objecter ms_dispatch 0x7f3e00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:20.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.338+0000 7f3dfcff9640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:20.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.338+0000 7f3dfcff9640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:20.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.339+0000 7f3dfcff9640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T19:23:20.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.339+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3df80504e0 con 0x7f3e001430c0 2024-03-25T19:23:20.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.339+0000 7f3dfcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:20.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.339+0000 7f3dfcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:20.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.592+0000 7f3e05c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:20.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.592+0000 7f3e05c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:20.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.592+0000 7f3e05c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:23:20.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.592+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e00074cc0 con 0x7f3e001430c0 2024-03-25T19:23:20.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.734+0000 7f3dfcff9640 1 -- 172.21.15.60:0/3714770075 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f3df8052410 con 0x7f3e001430c0 2024-03-25T19:23:20.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.734+0000 7f3dfcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:20.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.734+0000 7f3dfcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:20.735 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:20.735 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 10 librados: watch_flush enter 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 10 librados: watch_flush exit 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 10 client.4346.objecter successfully canceled tick 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 >> v1:172.21.15.90:6800/1827638754 conn(0x7f3de006de60 legacy=0x7f3de0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 10 monclient: shutdown 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.736+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 >> v1:172.21.15.60:6790/0 conn(0x7f3e001430c0 legacy=0x7f3e001e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.737+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 shutdown_connections 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.737+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 >> 172.21.15.60:0/3714770075 conn(0x7f3e000b7ea0 msgr2=0x7f3e00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:20.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.737+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 shutdown_connections 2024-03-25T19:23:20.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.737+0000 7f3e05c0f640 1 -- 172.21.15.60:0/3714770075 wait complete. 2024-03-25T19:23:20.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:20.737+0000 7f3e05c0f640 1 librados: shutdown 2024-03-25T19:23:20.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:27.554 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.663+0000 7f233b2d9640 10 monclient: get_monmap_and_config 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.663+0000 7f233b2d9640 10 monclient: build_initial_monmap 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.663+0000 7f233b2d9640 10 monclient: monmap: 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:27.665424+0000 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:27.665424+0000 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:27.664 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:27.665 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:27.665 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:27.665 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:27.665 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:27.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.664+0000 7f233b2d9640 1 Processor -- start 2024-03-25T19:23:27.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.664+0000 7f233b2d9640 1 -- start start 2024-03-25T19:23:27.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.664+0000 7f233b2d9640 10 monclient: init 2024-03-25T19:23:27.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:27.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:23:27.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 10 monclient(hunting): picked mon.noname-a con 0x7f23340b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:27.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 10 monclient(hunting): picked mon.noname-b con 0x7f23340b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:27.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 10 monclient(hunting): picked mon.noname-c con 0x7f233414be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:27.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23340af940 con 0x7f23340b4d60 2024-03-25T19:23:27.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23340b0a70 con 0x7f233414be90 2024-03-25T19:23:27.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23340b0d50 con 0x7f23340b1730 2024-03-25T19:23:27.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:27.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.665+0000 7f233b2d9640 10 monclient(hunting): authenticate will time out at 6534.764657s 2024-03-25T19:23:27.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.673+0000 7f2339876640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f233414be90 0x7f233414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:35680/0 (socket says 172.21.15.60:35680) 2024-03-25T19:23:27.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.673+0000 7f2339876640 1 -- 172.21.15.60:0/3418133718 learned_addr learned my addr 172.21.15.60:0/3418133718 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:27.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.692+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38204458 0 0) 0x7f23340b0a70 con 0x7f233414be90 2024-03-25T19:23:27.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.692+0000 7f232bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:27.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.692+0000 7f232bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:27.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.692+0000 7f232bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:27.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.692+0000 7f232bfff640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:23:27.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.693+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2320003520 con 0x7f233414be90 2024-03-25T19:23:27.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.695+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027655544 0 0) 0x7f23340b0d50 con 0x7f23340b1730 2024-03-25T19:23:27.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.695+0000 7f232bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:27.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.695+0000 7f232bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:27.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.695+0000 7f232bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:27.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.695+0000 7f232bfff640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:23:27.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.695+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23340b0a70 con 0x7f23340b1730 2024-03-25T19:23:27.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2018941282 0 0) 0x7f2320003520 con 0x7f233414be90 2024-03-25T19:23:27.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 >> v1:172.21.15.60:6790/0 conn(0x7f23340b1730 legacy=0x7f23340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:27.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 >> v1:172.21.15.60:6789/0 conn(0x7f23340b4d60 legacy=0x7f233413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:27.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 1 monclient: found mon.noname-c 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23340b1030 con 0x7f233414be90 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 10 monclient: _finish_auth 0 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f232c001a10 con 0x7f233414be90 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:27.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f232bfff640 10 monclient: dump: 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:27.715 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f233b2d9640 5 monclient: authenticate success, global_id 4354 2024-03-25T19:23:27.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.713+0000 7f233b2d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f232c001b70 con 0x7f233414be90 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f232bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f232bfff640 1 -- 172.21.15.60:0/3418133718 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f232c001f80 con 0x7f233414be90 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f233b2d9640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f233b2d9640 4 set_mon_vals no callback set 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f232bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f232bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.728+0000 7f232bfff640 10 monclient: dump: 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:27.729 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:27.730 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:27.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.729+0000 7f233b2d9640 10 monclient: shutdown 2024-03-25T19:23:27.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.729+0000 7f233b2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:27.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.729+0000 7f233b2d9640 1 -- 172.21.15.60:0/3418133718 >> v1:172.21.15.90:6789/0 conn(0x7f233414be90 legacy=0x7f233414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:27.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.729+0000 7f233b2d9640 1 -- 172.21.15.60:0/3418133718 shutdown_connections 2024-03-25T19:23:27.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.729+0000 7f233b2d9640 1 -- 172.21.15.60:0/3418133718 >> 172.21.15.60:0/3418133718 conn(0x7f23340617a0 msgr2=0x7f2334061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.729+0000 7f233b2d9640 1 -- 172.21.15.60:0/3418133718 shutdown_connections 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.729+0000 7f233b2d9640 1 -- 172.21.15.60:0/3418133718 wait complete. 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.730+0000 7f233b2d9640 10 monclient: build_initial_monmap 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.730+0000 7f233b2d9640 10 monclient: monmap: 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:27.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.730+0000 7f233b2d9640 1 librados: starting msgr at 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.730+0000 7f233b2d9640 1 librados: starting objecter 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 1 Processor -- start 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 1 -- start start 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 1 librados: setting wanted keys 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 1 librados: calling monclient init 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient: init 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient(hunting): picked mon.noname-a con 0x7f23340b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient(hunting): picked mon.noname-b con 0x7f23340b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient(hunting): picked mon.noname-c con 0x7f233414be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2334134f90 con 0x7f23340b1730 2024-03-25T19:23:27.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2334139ee0 con 0x7f23340b4d60 2024-03-25T19:23:27.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233413a1c0 con 0x7f233414be90 2024-03-25T19:23:27.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:27.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.731+0000 7f233b2d9640 10 monclient(hunting): authenticate will time out at 6534.830775s 2024-03-25T19:23:27.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.735+0000 7f2339075640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f23340b1730 0x7f2334140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38320/0 (socket says 172.21.15.60:38320) 2024-03-25T19:23:27.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.735+0000 7f2339075640 1 -- 172.21.15.60:0/60732740 learned_addr learned my addr 172.21.15.60:0/60732740 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:27.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.735+0000 7f2329ffb640 10 client.?.objecter ms_handle_connect 0x7f23340b1730 2024-03-25T19:23:27.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.735+0000 7f2329ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:27.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.746+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996220867 0 0) 0x7f2334134f90 con 0x7f23340b1730 2024-03-25T19:23:27.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.746+0000 7f2329ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:27.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.746+0000 7f2329ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:27.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.746+0000 7f2329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:27.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.746+0000 7f2329ffb640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:23:27.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.746+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2318004eb0 con 0x7f23340b1730 2024-03-25T19:23:27.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.753+0000 7f2329ffb640 10 client.?.objecter ms_handle_connect 0x7f233414be90 2024-03-25T19:23:27.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.753+0000 7f2329ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:27.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.755+0000 7f2329ffb640 10 client.?.objecter ms_handle_connect 0x7f23340b4d60 2024-03-25T19:23:27.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.755+0000 7f2329ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:27.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.768+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153974199 0 0) 0x7f2334139ee0 con 0x7f23340b4d60 2024-03-25T19:23:27.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.768+0000 7f2329ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:27.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.768+0000 7f2329ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:27.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.768+0000 7f2329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:27.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.768+0000 7f2329ffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:23:27.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.768+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2334134f90 con 0x7f23340b4d60 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.770+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635605394 0 0) 0x7f2318004eb0 con 0x7f23340b1730 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.770+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2334139ee0 con 0x7f23340b1730 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.770+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f23300019d0 con 0x7f23340b1730 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.770+0000 7f2329ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.770+0000 7f2329ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.770+0000 7f2329ffb640 10 monclient(hunting): dump: 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:27.771 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:27.772 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:27.772 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:27.772 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:27.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.770+0000 7f2329ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:27.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737674534 0 0) 0x7f233413a1c0 con 0x7f233414be90 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2318004eb0 con 0x7f233414be90 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 553108357 0 0) 0x7f2334139ee0 con 0x7f23340b1730 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 >> v1:172.21.15.60:6790/0 conn(0x7f233414be90 legacy=0x7f2334134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 >> v1:172.21.15.90:6789/0 conn(0x7f23340b4d60 legacy=0x7f2334134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 1 monclient: found mon.a 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:27.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2334142c40 con 0x7f23340b1730 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient: _finish_auth 0 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f2329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f233b2d9640 5 monclient: authenticate success, global_id 4365 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f233b2d9640 10 monclient: _renew_subs 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f233b2d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.779+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2334140a90 con 0x7f23340b1730 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.780+0000 7f233b2d9640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.780+0000 7f233b2d9640 10 monclient: _renew_subs 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.780+0000 7f233b2d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.780+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2334143320 con 0x7f23340b1730 2024-03-25T19:23:27.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.780+0000 7f233b2d9640 1 librados: init done 2024-03-25T19:23:27.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.780+0000 7f233b2d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:27.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.781+0000 7f233b2d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:27.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.781+0000 7f233b2d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:27.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:27.781+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23340627b0 con 0x7f23340b1730 2024-03-25T19:23:28.100 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.099+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2330003610 con 0x7f23340b1730 2024-03-25T19:23:28.100 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.099+0000 7f2329ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:28.100 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.099+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f23300039a0 con 0x7f23340b1730 2024-03-25T19:23:28.100 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.099+0000 7f2329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:28.100 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.099+0000 7f2329ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:28.100 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.099+0000 7f2329ffb640 10 monclient: dump: 2024-03-25T19:23:28.100 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:28.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.099+0000 7f232b7fe640 4 set_mon_vals no callback set 2024-03-25T19:23:28.102 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.100+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2330003bc0 con 0x7f23340b1730 2024-03-25T19:23:28.102 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.101+0000 7f2338874640 10 monclient: get_auth_request con 0x7f231806de00 auth_method 0 2024-03-25T19:23:28.102 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.101+0000 7f2329ffb640 10 client.4365.objecter ms_handle_connect 0x7f231806de00 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f23300547b0 con 0x7f23340b1730 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 10 client.4365.objecter ms_dispatch 0x7f2334000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23300502c0 con 0x7f23340b1730 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:28.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.107+0000 7f2329ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.366+0000 7f233b2d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.366+0000 7f233b2d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.366+0000 7f233b2d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.366+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2334142dd0 con 0x7f23340b1730 2024-03-25T19:23:28.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.397+0000 7f2329ffb640 1 -- 172.21.15.60:0/60732740 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f23300521f0 con 0x7f23340b1730 2024-03-25T19:23:28.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.397+0000 7f2329ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:28.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.397+0000 7f2329ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:28.398 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:28.398 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:28.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.399+0000 7f233b2d9640 10 librados: watch_flush enter 2024-03-25T19:23:28.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.399+0000 7f233b2d9640 10 librados: watch_flush exit 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.399+0000 7f233b2d9640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.399+0000 7f233b2d9640 10 client.4365.objecter successfully canceled tick 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 >> v1:172.21.15.90:6800/1827638754 conn(0x7f231806de00 legacy=0x7f23180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 10 monclient: shutdown 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 >> v1:172.21.15.60:6789/0 conn(0x7f23340b1730 legacy=0x7f2334140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 shutdown_connections 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 >> 172.21.15.60:0/60732740 conn(0x7f23340617a0 msgr2=0x7f23340b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 shutdown_connections 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 1 -- 172.21.15.60:0/60732740 wait complete. 2024-03-25T19:23:28.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:28.400+0000 7f233b2d9640 1 librados: shutdown 2024-03-25T19:23:28.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:34.211 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:34.922 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.256 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:34.910 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.258 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:34.912 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.258 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:34.970 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.260 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:34.972 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.263 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:34.975 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.272 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:34.926 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.275 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:34.986 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:34.285 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:34.996 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:35.217 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.329+0000 7f86c997c640 10 monclient: get_monmap_and_config 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.329+0000 7f86c997c640 10 monclient: build_initial_monmap 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.329+0000 7f86c997c640 10 monclient: monmap: 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:35.331387+0000 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:35.331387+0000 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:35.330 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:35.331 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 1 Processor -- start 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 1 -- start start 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient: init 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient(hunting): picked mon.noname-a con 0x7f86c40704a0 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient(hunting): picked mon.noname-c con 0x7f86c4070880 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient(hunting): picked mon.noname-b con 0x7f86c4149a40 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c4143870 con 0x7f86c40704a0 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c4072480 con 0x7f86c4070880 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c4072760 con 0x7f86c4149a40 2024-03-25T19:23:35.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:35.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86c997c640 10 monclient(hunting): authenticate will time out at 6542.429939s 2024-03-25T19:23:35.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86bbfff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f86c4070880 0x7f86c413d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:40580/0 (socket says 172.21.15.60:40580) 2024-03-25T19:23:35.335 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.331+0000 7f86bbfff640 1 -- 172.21.15.60:0/735828126 learned_addr learned my addr 172.21.15.60:0/735828126 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:35.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.362+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100732330 0 0) 0x7f86c4143870 con 0x7f86c40704a0 2024-03-25T19:23:35.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.362+0000 7f86c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:35.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.362+0000 7f86c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:35.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.362+0000 7f86c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:35.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.362+0000 7f86c27fc640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:23:35.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.362+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b0003520 con 0x7f86c40704a0 2024-03-25T19:23:35.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.365+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236968339 0 0) 0x7f86c4072480 con 0x7f86c4070880 2024-03-25T19:23:35.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.365+0000 7f86c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:35.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.365+0000 7f86c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:35.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.365+0000 7f86c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:35.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.365+0000 7f86c27fc640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:23:35.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.365+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4143870 con 0x7f86c4070880 2024-03-25T19:23:35.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.374+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783285561 0 0) 0x7f86c4072760 con 0x7f86c4149a40 2024-03-25T19:23:35.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.374+0000 7f86c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:23:35.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.374+0000 7f86c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:35.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.374+0000 7f86c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:35.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.374+0000 7f86c27fc640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:23:35.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.374+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4072480 con 0x7f86c4149a40 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715470047 0 0) 0x7f86b0003520 con 0x7f86c40704a0 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 >> v1:172.21.15.60:6790/0 conn(0x7f86c4149a40 legacy=0x7f86c414be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 >> v1:172.21.15.90:6789/0 conn(0x7f86c4070880 legacy=0x7f86c413d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 1 monclient: found mon.noname-a 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c4072a40 con 0x7f86c40704a0 2024-03-25T19:23:35.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 10 monclient: _finish_auth 0 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86b4001690 con 0x7f86c40704a0 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c27fc640 10 monclient: dump: 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:35.398 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:35.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c997c640 5 monclient: authenticate success, global_id 4368 2024-03-25T19:23:35.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.396+0000 7f86c997c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:35.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b40017f0 con 0x7f86c40704a0 2024-03-25T19:23:35.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:35.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c27fc640 1 -- 172.21.15.60:0/735828126 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86b4001b80 con 0x7f86c40704a0 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c27fc640 10 monclient: dump: 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c997c640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:35.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.467+0000 7f86c997c640 4 set_mon_vals no callback set 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.468+0000 7f86c997c640 10 monclient: shutdown 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.468+0000 7f86c997c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.468+0000 7f86c997c640 1 -- 172.21.15.60:0/735828126 >> v1:172.21.15.60:6789/0 conn(0x7f86c40704a0 legacy=0x7f86c4143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.468+0000 7f86c997c640 1 -- 172.21.15.60:0/735828126 shutdown_connections 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.468+0000 7f86c997c640 1 -- 172.21.15.60:0/735828126 >> 172.21.15.60:0/735828126 conn(0x7f86c40b5c00 msgr2=0x7f86c40b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.468+0000 7f86c997c640 1 -- 172.21.15.60:0/735828126 shutdown_connections 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.468+0000 7f86c997c640 1 -- 172.21.15.60:0/735828126 wait complete. 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 10 monclient: build_initial_monmap 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 10 monclient: monmap: 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:35.470 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 1 librados: starting msgr at 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 1 librados: starting objecter 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 1 Processor -- start 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 1 -- start start 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 1 librados: setting wanted keys 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 1 librados: calling monclient init 2024-03-25T19:23:35.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.469+0000 7f86c997c640 10 monclient: init 2024-03-25T19:23:35.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:35.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:35.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 10 monclient(hunting): picked mon.noname-c con 0x7f86c40704a0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:35.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 10 monclient(hunting): picked mon.noname-a con 0x7f86c4070880 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:35.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 10 monclient(hunting): picked mon.noname-b con 0x7f86c4149a40 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:35.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c413a1a0 con 0x7f86c4070880 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c413eab0 con 0x7f86c4149a40 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c4139030 con 0x7f86c40704a0 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.470+0000 7f86c997c640 10 monclient(hunting): authenticate will time out at 6542.569569s 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.471+0000 7f86c2ffd640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f86c40704a0 0x7f86c4139940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:50880/0 (socket says 172.21.15.60:50880) 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.471+0000 7f86c2ffd640 1 -- 172.21.15.60:0/3635186389 learned_addr learned my addr 172.21.15.60:0/3635186389 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.471+0000 7f86c0ff9640 10 client.?.objecter ms_handle_connect 0x7f86c40704a0 2024-03-25T19:23:35.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.471+0000 7f86c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:35.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.475+0000 7f86c0ff9640 10 client.?.objecter ms_handle_connect 0x7f86c4070880 2024-03-25T19:23:35.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.475+0000 7f86c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:35.481 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.480+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433391093 0 0) 0x7f86c413a1a0 con 0x7f86c4070880 2024-03-25T19:23:35.481 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.480+0000 7f86c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:35.481 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.480+0000 7f86c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:35.481 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.480+0000 7f86c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:35.481 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.480+0000 7f86c0ff9640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:23:35.481 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.480+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a4004eb0 con 0x7f86c4070880 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.492+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477975419 0 0) 0x7f86a4004eb0 con 0x7f86c4070880 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.492+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c413a1a0 con 0x7f86c4070880 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.492+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86ac001970 con 0x7f86c4070880 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.492+0000 7f86c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.492+0000 7f86c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.492+0000 7f86c0ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.493 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.494 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:35.494 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:35.494 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:35.494 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:35.494 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:35.494 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:35.494 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.492+0000 7f86c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1620221426 0 0) 0x7f86c413a1a0 con 0x7f86c4070880 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 >> v1:172.21.15.60:6790/0 conn(0x7f86c40704a0 legacy=0x7f86c4139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 >> v1:172.21.15.90:6789/0 conn(0x7f86c4149a40 legacy=0x7f86c4137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 1 monclient: found mon.a 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c41391c0 con 0x7f86c4070880 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 10 monclient: _renew_subs 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:35.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c4142660 con 0x7f86c4070880 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 10 monclient: _renew_subs 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c4140d30 con 0x7f86c4070880 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.511+0000 7f86c997c640 1 librados: init done 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.512+0000 7f86c997c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.512+0000 7f86c997c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.512+0000 7f86c997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:35.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.512+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c413a350 con 0x7f86c4070880 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.792+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86ac001c50 con 0x7f86c4070880 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.792+0000 7f86c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.792+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86ac004a90 con 0x7f86c4070880 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.792+0000 7f86c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.792+0000 7f86c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.792+0000 7f86c0ff9640 10 monclient: dump: 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:35.793 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:35.794 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:35.794 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:35.794 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:35.794 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:35.794 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:35.794 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.792+0000 7f86c1ffb640 4 set_mon_vals no callback set 2024-03-25T19:23:35.794 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.793+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f86ac01aa60 con 0x7f86c4070880 2024-03-25T19:23:35.795 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.793+0000 7f86c2ffd640 10 monclient: get_auth_request con 0x7f86a406dcb0 auth_method 0 2024-03-25T19:23:35.795 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.794+0000 7f86c0ff9640 10 client.4371.objecter ms_handle_connect 0x7f86a406dcb0 2024-03-25T19:23:35.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f86ac054720 con 0x7f86c4070880 2024-03-25T19:23:35.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 10 client.4371.objecter ms_dispatch 0x7f86c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:35.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:35.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:35.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T19:23:35.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86ac050060 con 0x7f86c4070880 2024-03-25T19:23:35.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:35.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:35.803+0000 7f86c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:36.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.062+0000 7f86c997c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:36.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.062+0000 7f86c997c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:36.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.062+0000 7f86c997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:23:36.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.062+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86c413a610 con 0x7f86c4070880 2024-03-25T19:23:36.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.094+0000 7f86c0ff9640 1 -- 172.21.15.60:0/3635186389 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f86c413a610 con 0x7f86c4070880 2024-03-25T19:23:36.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.094+0000 7f86c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:36.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.094+0000 7f86c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:36.093 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:36.094 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:36.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.096+0000 7f86c997c640 10 librados: watch_flush enter 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 10 librados: watch_flush exit 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 10 client.4371.objecter successfully canceled tick 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 >> v1:172.21.15.90:6800/1827638754 conn(0x7f86a406dcb0 legacy=0x7f86a4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 10 monclient: shutdown 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 >> v1:172.21.15.60:6789/0 conn(0x7f86c4070880 legacy=0x7f86c4140440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 shutdown_connections 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 >> 172.21.15.60:0/3635186389 conn(0x7f86c40b5c00 msgr2=0x7f86c40b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 shutdown_connections 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 1 -- 172.21.15.60:0/3635186389 wait complete. 2024-03-25T19:23:36.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:36.097+0000 7f86c997c640 1 librados: shutdown 2024-03-25T19:23:36.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:42.914 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:43.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.031+0000 7fa8b6ea8640 10 monclient: get_monmap_and_config 2024-03-25T19:23:43.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.031+0000 7fa8b6ea8640 10 monclient: build_initial_monmap 2024-03-25T19:23:43.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.031+0000 7fa8b6ea8640 10 monclient: monmap: 2024-03-25T19:23:43.031 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:43.031 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:43.031 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:43.032589+0000 2024-03-25T19:23:43.032 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:43.032589+0000 2024-03-25T19:23:43.032 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:43.032 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:43.032 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:43.032 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:43.032 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:43.032 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:43.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 1 Processor -- start 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 1 -- start start 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient: init 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b00705b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b01402d0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b0149ae0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00718c0 con 0x7fa8b00705b0 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00714c0 con 0x7fa8b01402d0 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b0072b50 con 0x7fa8b0149ae0 2024-03-25T19:23:43.034 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:43.035 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.033+0000 7fa8b6ea8640 10 monclient(hunting): authenticate will time out at 6550.131679s 2024-03-25T19:23:43.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.040+0000 7fa8b5445640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fa8b0149ae0 0x7fa8b014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:59066/0 (socket says 172.21.15.60:59066) 2024-03-25T19:23:43.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.040+0000 7fa8b5445640 1 -- 172.21.15.60:0/1613352122 learned_addr learned my addr 172.21.15.60:0/1613352122 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:43.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.049+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762293073 0 0) 0x7fa8b0072b50 con 0x7fa8b0149ae0 2024-03-25T19:23:43.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.049+0000 7fa8af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:43.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.049+0000 7fa8af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:43.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.049+0000 7fa8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:43.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.049+0000 7fa8af7fe640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:23:43.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.049+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a0003520 con 0x7fa8b0149ae0 2024-03-25T19:23:43.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.054+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278927949 0 0) 0x7fa8b00718c0 con 0x7fa8b00705b0 2024-03-25T19:23:43.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.054+0000 7fa8af7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:23:43.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.054+0000 7fa8af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:43.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.054+0000 7fa8af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:43.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.054+0000 7fa8af7fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:23:43.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.054+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b0072b50 con 0x7fa8b00705b0 2024-03-25T19:23:43.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1817121516 0 0) 0x7fa8a0003520 con 0x7fa8b0149ae0 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 >> v1:172.21.15.90:6789/0 conn(0x7fa8b01402d0 legacy=0x7fa8b0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 >> v1:172.21.15.60:6789/0 conn(0x7fa8b00705b0 legacy=0x7fa8b013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 1 monclient: found mon.noname-b 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b013dd10 con 0x7fa8b0149ae0 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 10 monclient: _finish_auth 0 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa8a8001a10 con 0x7fa8b0149ae0 2024-03-25T19:23:43.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8af7fe640 10 monclient: dump: 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8b6ea8640 5 monclient: authenticate success, global_id 4358 2024-03-25T19:23:43.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.056+0000 7fa8b6ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:43.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8a8001b70 con 0x7fa8b0149ae0 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8af7fe640 1 -- 172.21.15.60:0/1613352122 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa8a8001f80 con 0x7fa8b0149ae0 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8af7fe640 10 monclient: dump: 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:43.060 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:43.061 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:43.061 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:43.061 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:43.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8b6ea8640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:43.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.060+0000 7fa8b6ea8640 4 set_mon_vals no callback set 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.061+0000 7fa8b6ea8640 10 monclient: shutdown 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.061+0000 7fa8b6ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.061+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/1613352122 >> v1:172.21.15.60:6790/0 conn(0x7fa8b0149ae0 legacy=0x7fa8b014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.061+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/1613352122 shutdown_connections 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.061+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/1613352122 >> 172.21.15.60:0/1613352122 conn(0x7fa8b00b5d10 msgr2=0x7fa8b00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.061+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/1613352122 shutdown_connections 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.061+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/1613352122 wait complete. 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.062+0000 7fa8b6ea8640 10 monclient: build_initial_monmap 2024-03-25T19:23:43.063 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.062+0000 7fa8b6ea8640 10 monclient: monmap: 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.062+0000 7fa8b6ea8640 1 librados: starting msgr at 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.062+0000 7fa8b6ea8640 1 librados: starting objecter 2024-03-25T19:23:43.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.062+0000 7fa8b6ea8640 1 Processor -- start 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 1 -- start start 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 1 librados: setting wanted keys 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 1 librados: calling monclient init 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient: init 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b00705b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b01402d0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b0149ae0 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00a9fe0 con 0x7fa8b0149ae0 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00aa2c0 con 0x7fa8b01402d0 2024-03-25T19:23:43.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00aa5a0 con 0x7fa8b00705b0 2024-03-25T19:23:43.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:43.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.063+0000 7fa8b6ea8640 10 monclient(hunting): authenticate will time out at 6550.161785s 2024-03-25T19:23:43.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.075+0000 7fa8affff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa8b01402d0 0x7fa8b013e830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:36850/0 (socket says 172.21.15.60:36850) 2024-03-25T19:23:43.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.075+0000 7fa8affff640 1 -- 172.21.15.60:0/862853204 learned_addr learned my addr 172.21.15.60:0/862853204 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:43.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.076+0000 7fa8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fa8b01402d0 2024-03-25T19:23:43.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.076+0000 7fa8ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:43.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.089+0000 7fa8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fa8b0149ae0 2024-03-25T19:23:43.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.089+0000 7fa8ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:43.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.094+0000 7fa8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fa8b00705b0 2024-03-25T19:23:43.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.094+0000 7fa8ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:43.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.096+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307495565 0 0) 0x7fa8b00aa2c0 con 0x7fa8b01402d0 2024-03-25T19:23:43.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.096+0000 7fa8ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:43.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.096+0000 7fa8ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:43.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.096+0000 7fa8ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:43.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.096+0000 7fa8ad7fa640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:23:43.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.096+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa894004eb0 con 0x7fa8b01402d0 2024-03-25T19:23:43.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.107+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242823353 0 0) 0x7fa8b00aa5a0 con 0x7fa8b00705b0 2024-03-25T19:23:43.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.107+0000 7fa8ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:43.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.107+0000 7fa8ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:43.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.107+0000 7fa8ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:43.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.107+0000 7fa8ad7fa640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:23:43.107 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.107+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b00aa2c0 con 0x7fa8b00705b0 2024-03-25T19:23:43.110 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.111+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3405896986 0 0) 0x7fa894004eb0 con 0x7fa8b01402d0 2024-03-25T19:23:43.110 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.111+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8b00aa5a0 con 0x7fa8b01402d0 2024-03-25T19:23:43.110 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.111+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa89c001a30 con 0x7fa8b01402d0 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.111+0000 7fa8ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.111+0000 7fa8ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.111+0000 7fa8ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:43.111 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:43.112 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:43.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.111+0000 7fa8ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:43.115 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.115+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709977839 0 0) 0x7fa8b00a9fe0 con 0x7fa8b0149ae0 2024-03-25T19:23:43.115 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.115+0000 7fa8ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:23:43.115 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.115+0000 7fa8ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:43.115 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.115+0000 7fa8ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:43.115 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.115+0000 7fa8ad7fa640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:23:43.115 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.115+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa894004eb0 con 0x7fa8b0149ae0 2024-03-25T19:23:43.137 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.137+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4268412258 0 0) 0x7fa8b00aa2c0 con 0x7fa8b00705b0 2024-03-25T19:23:43.137 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8b00a9fe0 con 0x7fa8b00705b0 2024-03-25T19:23:43.137 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa8a4001930 con 0x7fa8b00705b0 2024-03-25T19:23:43.137 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:43.137 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:43.137 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 10 monclient(hunting): dump: 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:43.138 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650164231 0 0) 0x7fa8b00aa5a0 con 0x7fa8b01402d0 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 >> v1:172.21.15.60:6790/0 conn(0x7fa8b00705b0 legacy=0x7fa8b01338e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 >> v1:172.21.15.60:6789/0 conn(0x7fa8b0149ae0 legacy=0x7fa8b013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 1 monclient: found mon.b 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b01345c0 con 0x7fa8b01402d0 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 10 monclient: _finish_auth 0 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:23:43.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 10 monclient: _renew_subs 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b00a96f0 con 0x7fa8b01402d0 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 10 monclient: _renew_subs 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b006d860 con 0x7fa8b01402d0 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.138+0000 7fa8b6ea8640 1 librados: init done 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.139+0000 7fa8b6ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.139+0000 7fa8b6ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.139+0000 7fa8b6ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:43.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.139+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b00627b0 con 0x7fa8b01402d0 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa89c001c70 con 0x7fa8b01402d0 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa89c003a60 con 0x7fa8b01402d0 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8ad7fa640 10 monclient: dump: 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:43.467 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8aeffd640 4 set_mon_vals no callback set 2024-03-25T19:23:43.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.467+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fa89c003c80 con 0x7fa8b01402d0 2024-03-25T19:23:43.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.468+0000 7fa8b4c44640 10 monclient: get_auth_request con 0x7fa89406dfd0 auth_method 0 2024-03-25T19:23:43.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.468+0000 7fa8ad7fa640 10 client.4366.objecter ms_handle_connect 0x7fa89406dfd0 2024-03-25T19:23:43.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.472+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fa89c054870 con 0x7fa8b01402d0 2024-03-25T19:23:43.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.472+0000 7fa8ad7fa640 10 client.4366.objecter ms_dispatch 0x7fa8b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:43.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.472+0000 7fa8ad7fa640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:43.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.472+0000 7fa8ad7fa640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:43.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.472+0000 7fa8ad7fa640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:23:43.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.473+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa89c050380 con 0x7fa8b01402d0 2024-03-25T19:23:43.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.473+0000 7fa8ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:43.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.473+0000 7fa8ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:43.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.708+0000 7fa8b6ea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:43.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.708+0000 7fa8b6ea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:43.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.708+0000 7fa8b6ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:43.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.708+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8b013e550 con 0x7fa8b01402d0 2024-03-25T19:23:43.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.745+0000 7fa8ad7fa640 1 -- 172.21.15.60:0/862853204 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fa8b013e550 con 0x7fa8b01402d0 2024-03-25T19:23:43.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.745+0000 7fa8ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:43.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.745+0000 7fa8ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:43.745 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:43.745 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:43.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 10 librados: watch_flush enter 2024-03-25T19:23:43.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 10 librados: watch_flush exit 2024-03-25T19:23:43.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:23:43.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:23:43.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 >> v1:172.21.15.90:6800/1827638754 conn(0x7fa89406dfd0 legacy=0x7fa894070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 10 monclient: shutdown 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 >> v1:172.21.15.90:6789/0 conn(0x7fa8b01402d0 legacy=0x7fa8b013e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 shutdown_connections 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 >> 172.21.15.60:0/862853204 conn(0x7fa8b00b5d10 msgr2=0x7fa8b0140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 shutdown_connections 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 1 -- 172.21.15.60:0/862853204 wait complete. 2024-03-25T19:23:43.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:43.748+0000 7fa8b6ea8640 1 librados: shutdown 2024-03-25T19:23:43.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:50.567 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:50.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.695+0000 7f276e2d4640 10 monclient: get_monmap_and_config 2024-03-25T19:23:50.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.695+0000 7f276e2d4640 10 monclient: build_initial_monmap 2024-03-25T19:23:50.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.695+0000 7f276e2d4640 10 monclient: monmap: 2024-03-25T19:23:50.695 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:50.695 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:50.695 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:50.696443+0000 2024-03-25T19:23:50.695 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:50.696443+0000 2024-03-25T19:23:50.696 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:50.696 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:50.696 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:50.696 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:50.696 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:50.696 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:50.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.696+0000 7f276e2d4640 1 Processor -- start 2024-03-25T19:23:50.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.696+0000 7f276e2d4640 1 -- start start 2024-03-25T19:23:50.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.696+0000 7f276e2d4640 10 monclient: init 2024-03-25T19:23:50.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:50.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:23:50.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 10 monclient(hunting): picked mon.noname-c con 0x7f27680a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:50.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 10 monclient(hunting): picked mon.noname-b con 0x7f276813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:50.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 10 monclient(hunting): picked mon.noname-a con 0x7f27681430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:50.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2768142520 con 0x7f27681430c0 2024-03-25T19:23:50.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27680b28a0 con 0x7f27680a98f0 2024-03-25T19:23:50.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27680b2a30 con 0x7f276813f710 2024-03-25T19:23:50.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:50.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.697+0000 7f276e2d4640 10 monclient(hunting): authenticate will time out at 6557.795818s 2024-03-25T19:23:50.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.702+0000 7f276c871640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f27681430c0 0x7f276814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:59338/0 (socket says 172.21.15.60:59338) 2024-03-25T19:23:50.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.702+0000 7f276c871640 1 -- 172.21.15.60:0/4082657037 learned_addr learned my addr 172.21.15.60:0/4082657037 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:50.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.720+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035526556 0 0) 0x7f2768142520 con 0x7f27681430c0 2024-03-25T19:23:50.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.720+0000 7f2766ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:50.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.720+0000 7f2766ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:50.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.720+0000 7f2766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:50.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.720+0000 7f2766ffd640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:23:50.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.720+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2750003520 con 0x7f27681430c0 2024-03-25T19:23:50.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.722+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060500313 0 0) 0x7f27680b28a0 con 0x7f27680a98f0 2024-03-25T19:23:50.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.722+0000 7f2766ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:50.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.722+0000 7f2766ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:50.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.722+0000 7f2766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:50.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.722+0000 7f2766ffd640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:23:50.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.722+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2768142520 con 0x7f27680a98f0 2024-03-25T19:23:50.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.733+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966204608 0 0) 0x7f27680b2a30 con 0x7f276813f710 2024-03-25T19:23:50.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.733+0000 7f2766ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:23:50.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.733+0000 7f2766ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:50.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.733+0000 7f2766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:50.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.733+0000 7f2766ffd640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:23:50.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.733+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27680b28a0 con 0x7f276813f710 2024-03-25T19:23:50.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563550997 0 0) 0x7f2768142520 con 0x7f27680a98f0 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 >> v1:172.21.15.60:6790/0 conn(0x7f276813f710 legacy=0x7f27681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 >> v1:172.21.15.60:6789/0 conn(0x7f27681430c0 legacy=0x7f276814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 1 monclient: found mon.noname-c 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27680b44c0 con 0x7f27680a98f0 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 10 monclient: _finish_auth 0 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2758001670 con 0x7f27680a98f0 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f276e2d4640 5 monclient: authenticate success, global_id 4369 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f276e2d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:50.739 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f2766ffd640 10 monclient: dump: 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:50.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.739+0000 7f276e2d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27580017d0 con 0x7f27680a98f0 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f2766ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f2766ffd640 1 -- 172.21.15.60:0/4082657037 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2758001be0 con 0x7f27680a98f0 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f276e2d4640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f276e2d4640 4 set_mon_vals no callback set 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f2766ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f2766ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.745+0000 7f2766ffd640 10 monclient: dump: 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.745 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.746 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:50.746 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:50.746 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:50.746 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:50.746 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:50.746 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:50.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.746+0000 7f276e2d4640 10 monclient: shutdown 2024-03-25T19:23:50.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.746+0000 7f276e2d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:50.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.746+0000 7f276e2d4640 1 -- 172.21.15.60:0/4082657037 >> v1:172.21.15.90:6789/0 conn(0x7f27680a98f0 legacy=0x7f27680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:50.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.746+0000 7f276e2d4640 1 -- 172.21.15.60:0/4082657037 shutdown_connections 2024-03-25T19:23:50.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.746+0000 7f276e2d4640 1 -- 172.21.15.60:0/4082657037 >> 172.21.15.60:0/4082657037 conn(0x7f27680b7ea0 msgr2=0x7f27680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:50.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.746+0000 7f276e2d4640 1 -- 172.21.15.60:0/4082657037 shutdown_connections 2024-03-25T19:23:50.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.746+0000 7f276e2d4640 1 -- 172.21.15.60:0/4082657037 wait complete. 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.747+0000 7f276e2d4640 10 monclient: build_initial_monmap 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.747+0000 7f276e2d4640 10 monclient: monmap: 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.747+0000 7f276e2d4640 1 librados: starting msgr at 2024-03-25T19:23:50.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.747+0000 7f276e2d4640 1 librados: starting objecter 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.747+0000 7f276e2d4640 1 Processor -- start 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 1 -- start start 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 1 librados: setting wanted keys 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 1 librados: calling monclient init 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient: init 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient(hunting): picked mon.noname-c con 0x7f27680a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient(hunting): picked mon.noname-a con 0x7f276813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient(hunting): picked mon.noname-b con 0x7f27681430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27681db820 con 0x7f276813f710 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276813a1b0 con 0x7f27681430c0 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276813ad80 con 0x7f27680a98f0 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:50.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.748+0000 7f276e2d4640 10 monclient(hunting): authenticate will time out at 6557.846603s 2024-03-25T19:23:50.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.763+0000 7f276c871640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f27681430c0 0x7f27681db110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:49862/0 (socket says 172.21.15.60:49862) 2024-03-25T19:23:50.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.763+0000 7f276c871640 1 -- 172.21.15.60:0/3885575559 learned_addr learned my addr 172.21.15.60:0/3885575559 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:50.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.764+0000 7f2764ff9640 10 client.?.objecter ms_handle_connect 0x7f27681430c0 2024-03-25T19:23:50.763 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.764+0000 7f2764ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:50.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.768+0000 7f2764ff9640 10 client.?.objecter ms_handle_connect 0x7f27680a98f0 2024-03-25T19:23:50.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.768+0000 7f2764ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:50.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.770+0000 7f2764ff9640 10 client.?.objecter ms_handle_connect 0x7f276813f710 2024-03-25T19:23:50.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.770+0000 7f2764ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:50.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.777+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932296476 0 0) 0x7f276813a1b0 con 0x7f27681430c0 2024-03-25T19:23:50.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.777+0000 7f2764ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:50.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.777+0000 7f2764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:50.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.777+0000 7f2764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:50.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.777+0000 7f2764ff9640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:23:50.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.777+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2754004eb0 con 0x7f27681430c0 2024-03-25T19:23:50.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.787+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864797878 0 0) 0x7f276813ad80 con 0x7f27680a98f0 2024-03-25T19:23:50.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.787+0000 7f2764ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:50.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.787+0000 7f2764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:50.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.787+0000 7f2764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:50.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.787+0000 7f2764ff9640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:23:50.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.787+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f276813a1b0 con 0x7f27680a98f0 2024-03-25T19:23:50.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.791+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306367689 0 0) 0x7f27681db820 con 0x7f276813f710 2024-03-25T19:23:50.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.791+0000 7f2764ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:23:50.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.791+0000 7f2764ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:50.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.791+0000 7f2764ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:50.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.791+0000 7f2764ff9640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:23:50.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.791+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f276813ad80 con 0x7f276813f710 2024-03-25T19:23:50.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.804+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2373192061 0 0) 0x7f2754004eb0 con 0x7f27681430c0 2024-03-25T19:23:50.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.804+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27681db820 con 0x7f27681430c0 2024-03-25T19:23:50.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.804+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2760001b30 con 0x7f27681430c0 2024-03-25T19:23:50.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.804+0000 7f2764ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:50.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.804+0000 7f2764ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.804+0000 7f2764ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:50.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.804+0000 7f2764ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:50.811 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.812+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1966707785 0 0) 0x7f276813a1b0 con 0x7f27680a98f0 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.812+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2754004eb0 con 0x7f27680a98f0 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.812+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2758001ae0 con 0x7f27680a98f0 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.812+0000 7f2764ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.812+0000 7f2764ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.812+0000 7f2764ff9640 10 monclient(hunting): dump: 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:50.812 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:50.813 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:50.813 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:50.813 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 10406920 0 0) 0x7f27681db820 con 0x7f27681430c0 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 >> v1:172.21.15.60:6790/0 conn(0x7f27680a98f0 legacy=0x7f276812f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 >> v1:172.21.15.60:6789/0 conn(0x7f276813f710 legacy=0x7f27681daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 1 monclient: found mon.b 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276813b060 con 0x7f27681430c0 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 10 monclient: _finish_auth 0 2024-03-25T19:23:50.821 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f2764ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 10 monclient: _renew_subs 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276813b340 con 0x7f27681430c0 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 10 monclient: _renew_subs 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f276813b7d0 con 0x7f27681430c0 2024-03-25T19:23:50.822 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.821+0000 7f276e2d4640 1 librados: init done 2024-03-25T19:23:50.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.822+0000 7f276e2d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:50.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.822+0000 7f276e2d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:50.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.822+0000 7f276e2d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:50.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:50.822+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27680b3ef0 con 0x7f27681430c0 2024-03-25T19:23:51.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2760001fe0 con 0x7f27681430c0 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f2764ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2760004d00 con 0x7f27681430c0 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f2764ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f2764ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f2764ff9640 10 monclient: dump: 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:51.096 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:51.097 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:51.097 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:51.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f27667fc640 4 set_mon_vals no callback set 2024-03-25T19:23:51.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.096+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f276001acd0 con 0x7f27681430c0 2024-03-25T19:23:51.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.097+0000 7f2767fff640 10 monclient: get_auth_request con 0x7f275406df70 auth_method 0 2024-03-25T19:23:51.097 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.097+0000 7f2764ff9640 10 client.4372.objecter ms_handle_connect 0x7f275406df70 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f2760054a90 con 0x7f27681430c0 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 10 client.4372.objecter ms_dispatch 0x7f2768000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27600503d0 con 0x7f27681430c0 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:51.101 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.101+0000 7f2764ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:51.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.363+0000 7f276e2d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:51.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.363+0000 7f276e2d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:51.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.363+0000 7f276e2d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:51.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.363+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2768139d20 con 0x7f27681430c0 2024-03-25T19:23:51.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.408+0000 7f2764ff9640 1 -- 172.21.15.60:0/3885575559 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f27600524d0 con 0x7f27681430c0 2024-03-25T19:23:51.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.408+0000 7f2764ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:51.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.408+0000 7f2764ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:51.408 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:51.408 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:51.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 10 librados: watch_flush enter 2024-03-25T19:23:51.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 10 librados: watch_flush exit 2024-03-25T19:23:51.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T19:23:51.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 10 client.4372.objecter successfully canceled tick 2024-03-25T19:23:51.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 >> v1:172.21.15.90:6800/1827638754 conn(0x7f275406df70 legacy=0x7f2754070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 10 monclient: shutdown 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 >> v1:172.21.15.90:6789/0 conn(0x7f27681430c0 legacy=0x7f27681db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 shutdown_connections 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 >> 172.21.15.60:0/3885575559 conn(0x7f27680b7ea0 msgr2=0x7f2768074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 shutdown_connections 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.410+0000 7f276e2d4640 1 -- 172.21.15.60:0/3885575559 wait complete. 2024-03-25T19:23:51.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:51.411+0000 7f276e2d4640 1 librados: shutdown 2024-03-25T19:23:51.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:23:54.310 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:55.021 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.313 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:55.025 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.347 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:03:55.059 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.369 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:55.081 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.375 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:55.029 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.380 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:55.034 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.384 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:55.096 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.423 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:03:55.077 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:54.435 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:03:55.146 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:23:58.229 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:23:58.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.345+0000 7fb253886640 10 monclient: get_monmap_and_config 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.345+0000 7fb253886640 10 monclient: build_initial_monmap 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.345+0000 7fb253886640 10 monclient: monmap: 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:23:58.346078+0000 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:23:58.346078+0000 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:58.345 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:58.346 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:58.346 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:23:58.346 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:23:58.346 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 1 Processor -- start 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 1 -- start start 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 10 monclient: init 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:58.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:58.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.346+0000 7fb253886640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0af940 con 0x7fb24c0b1730 2024-03-25T19:23:58.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.347+0000 7fb253886640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0b0a70 con 0x7fb24c14be90 2024-03-25T19:23:58.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.347+0000 7fb253886640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0b0d50 con 0x7fb24c0b4d60 2024-03-25T19:23:58.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.347+0000 7fb253886640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:58.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.347+0000 7fb253886640 10 monclient(hunting): authenticate will time out at 6565.444916s 2024-03-25T19:23:58.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.364+0000 7fb251e23640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb24c14be90 0x7fb24c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:49230/0 (socket says 172.21.15.60:49230) 2024-03-25T19:23:58.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.364+0000 7fb251e23640 1 -- 172.21.15.60:0/793699900 learned_addr learned my addr 172.21.15.60:0/793699900 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:58.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.381+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956526230 0 0) 0x7fb24c0b0d50 con 0x7fb24c0b4d60 2024-03-25T19:23:58.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.381+0000 7fb243fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:58.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.381+0000 7fb243fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:58.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.381+0000 7fb243fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:58.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.381+0000 7fb243fff640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:23:58.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.381+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb238003520 con 0x7fb24c0b4d60 2024-03-25T19:23:58.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.383+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761391091 0 0) 0x7fb24c0b0a70 con 0x7fb24c14be90 2024-03-25T19:23:58.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.383+0000 7fb243fff640 10 monclient(hunting): _negotiate 2024-03-25T19:23:58.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.383+0000 7fb243fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:58.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.383+0000 7fb243fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:58.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.383+0000 7fb243fff640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:23:58.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.383+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c0b0d50 con 0x7fb24c14be90 2024-03-25T19:23:58.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.393+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1668850443 0 0) 0x7fb238003520 con 0x7fb24c0b4d60 2024-03-25T19:23:58.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 >> v1:172.21.15.90:6789/0 conn(0x7fb24c14be90 legacy=0x7fb24c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 >> v1:172.21.15.60:6789/0 conn(0x7fb24c0b1730 legacy=0x7fb24c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 10 monclient: _finish_hunting 0 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 1 monclient: found mon.noname-b 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c0b1030 con 0x7fb24c0b4d60 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 10 monclient: _finish_auth 0 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 10 monclient: _check_auth_tickets 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb248001690 con 0x7fb24c0b4d60 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb253886640 5 monclient: authenticate success, global_id 4370 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb253886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:58.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb243fff640 10 monclient: dump: 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:58.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.394+0000 7fb253886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:23:58.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2480017f0 con 0x7fb24c0b4d60 2024-03-25T19:23:58.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb243fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:58.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb243fff640 1 -- 172.21.15.60:0/793699900 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb248001c00 con 0x7fb24c0b4d60 2024-03-25T19:23:58.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb243fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:58.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb243fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:58.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb243fff640 10 monclient: dump: 2024-03-25T19:23:58.419 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb253886640 10 monclient: get_monmap_and_config success 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.417+0000 7fb253886640 4 set_mon_vals no callback set 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.418+0000 7fb253886640 10 monclient: shutdown 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.418+0000 7fb253886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:58.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.418+0000 7fb253886640 1 -- 172.21.15.60:0/793699900 >> v1:172.21.15.60:6790/0 conn(0x7fb24c0b4d60 legacy=0x7fb24c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.418+0000 7fb253886640 1 -- 172.21.15.60:0/793699900 shutdown_connections 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.418+0000 7fb253886640 1 -- 172.21.15.60:0/793699900 >> 172.21.15.60:0/793699900 conn(0x7fb24c0617a0 msgr2=0x7fb24c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.418+0000 7fb253886640 1 -- 172.21.15.60:0/793699900 shutdown_connections 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.418+0000 7fb253886640 1 -- 172.21.15.60:0/793699900 wait complete. 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 10 monclient: build_initial_monmap 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 10 monclient: monmap: 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:23:58.421 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 1 librados: starting msgr at 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 1 librados: starting objecter 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 1 Processor -- start 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 1 -- start start 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 1 librados: setting wanted keys 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 1 librados: calling monclient init 2024-03-25T19:23:58.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.419+0000 7fb253886640 10 monclient: init 2024-03-25T19:23:58.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 10 monclient: _reopen_session rank -1 2024-03-25T19:23:58.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:23:58.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 10 monclient(hunting): picked mon.noname-b con 0x7fb24c0b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:23:58.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 10 monclient(hunting): picked mon.noname-a con 0x7fb24c0b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:23:58.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 10 monclient(hunting): picked mon.noname-c con 0x7fb24c14be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:23:58.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c141bf0 con 0x7fb24c0b4d60 2024-03-25T19:23:58.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c141ed0 con 0x7fb24c0b1730 2024-03-25T19:23:58.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c1421b0 con 0x7fb24c14be90 2024-03-25T19:23:58.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 10 monclient(hunting): _renew_subs 2024-03-25T19:23:58.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.420+0000 7fb253886640 10 monclient(hunting): authenticate will time out at 6565.518643s 2024-03-25T19:23:58.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.429+0000 7fb251622640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb24c0b1730 0x7fb24c1425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:49302/0 (socket says 172.21.15.60:49302) 2024-03-25T19:23:58.428 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.429+0000 7fb251622640 1 -- 172.21.15.60:0/4101823333 learned_addr learned my addr 172.21.15.60:0/4101823333 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:23:58.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.431+0000 7fb241ffb640 10 client.?.objecter ms_handle_connect 0x7fb24c0b1730 2024-03-25T19:23:58.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.431+0000 7fb241ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:58.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.440+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834463463 0 0) 0x7fb24c141ed0 con 0x7fb24c0b1730 2024-03-25T19:23:58.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.440+0000 7fb241ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:58.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.440+0000 7fb241ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:58.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.440+0000 7fb241ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:58.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.440+0000 7fb241ffb640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:23:58.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.440+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb230004eb0 con 0x7fb24c0b1730 2024-03-25T19:23:58.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.441+0000 7fb241ffb640 10 client.?.objecter ms_handle_connect 0x7fb24c0b4d60 2024-03-25T19:23:58.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.441+0000 7fb241ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:58.444 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.445+0000 7fb241ffb640 10 client.?.objecter ms_handle_connect 0x7fb24c14be90 2024-03-25T19:23:58.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.445+0000 7fb241ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:23:58.451 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.452+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394233877 0 0) 0x7fb24c1421b0 con 0x7fb24c14be90 2024-03-25T19:23:58.451 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.452+0000 7fb241ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:58.451 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.452+0000 7fb241ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:58.451 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.452+0000 7fb241ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:58.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.452+0000 7fb241ffb640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:23:58.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.452+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb24c141ed0 con 0x7fb24c14be90 2024-03-25T19:23:58.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.457+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 464130916 0 0) 0x7fb230004eb0 con 0x7fb24c0b1730 2024-03-25T19:23:58.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.457+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24c1421b0 con 0x7fb24c0b1730 2024-03-25T19:23:58.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.457+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb248001690 con 0x7fb24c0b1730 2024-03-25T19:23:58.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.457+0000 7fb241ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.457+0000 7fb241ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.457+0000 7fb241ffb640 10 monclient(hunting): dump: 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:58.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.457+0000 7fb241ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:23:58.462 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.463+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113226966 0 0) 0x7fb24c141bf0 con 0x7fb24c0b4d60 2024-03-25T19:23:58.462 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.463+0000 7fb241ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:23:58.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.463+0000 7fb241ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:23:58.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.463+0000 7fb241ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:23:58.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.463+0000 7fb241ffb640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:23:58.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.463+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb230004eb0 con 0x7fb24c0b4d60 2024-03-25T19:23:58.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.465+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2370455614 0 0) 0x7fb24c141ed0 con 0x7fb24c14be90 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.465+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb24c141bf0 con 0x7fb24c14be90 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.465+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb244001b90 con 0x7fb24c14be90 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.465+0000 7fb241ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.465+0000 7fb241ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.465+0000 7fb241ffb640 10 monclient(hunting): dump: 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:58.465 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:58.466 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:58.466 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:58.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.471+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2086708123 0 0) 0x7fb24c1421b0 con 0x7fb24c0b1730 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 >> v1:172.21.15.60:6790/0 conn(0x7fb24c14be90 legacy=0x7fb24c1341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 >> v1:172.21.15.60:6789/0 conn(0x7fb24c0b4d60 legacy=0x7fb24c133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 1 monclient: found mon.b 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c140f00 con 0x7fb24c0b1730 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 10 monclient: _finish_auth 0 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb241ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 5 monclient: authenticate success, global_id 4378 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 10 monclient: _renew_subs 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:58.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c1406f0 con 0x7fb24c0b1730 2024-03-25T19:23:58.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:23:58.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 10 monclient: _renew_subs 2024-03-25T19:23:58.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:58.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c140380 con 0x7fb24c0b1730 2024-03-25T19:23:58.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.472+0000 7fb253886640 1 librados: init done 2024-03-25T19:23:58.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.473+0000 7fb253886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:58.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.473+0000 7fb253886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:58.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.473+0000 7fb253886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:58.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.473+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c1401f0 con 0x7fb24c0b1730 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2480038a0 con 0x7fb24c0b1730 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb241ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb248003c70 con 0x7fb24c0b1730 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb241ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb241ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb241ffb640 10 monclient: dump: 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:23:58.770 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:23:58.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb2437fe640 4 set_mon_vals no callback set 2024-03-25T19:23:58.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.770+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fb248003e90 con 0x7fb24c0b1730 2024-03-25T19:23:58.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.771+0000 7fb250e21640 10 monclient: get_auth_request con 0x7fb23006dfb0 auth_method 0 2024-03-25T19:23:58.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.771+0000 7fb241ffb640 10 client.4378.objecter ms_handle_connect 0x7fb23006dfb0 2024-03-25T19:23:58.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fb248054970 con 0x7fb24c0b1730 2024-03-25T19:23:58.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 10 client.4378.objecter ms_dispatch 0x7fb24c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:23:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:23:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:23:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T19:23:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb248050480 con 0x7fb24c0b1730 2024-03-25T19:23:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:23:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:58.777+0000 7fb241ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:23:59.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.014+0000 7fb253886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:59.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.014+0000 7fb253886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:59.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.014+0000 7fb253886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:23:59.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.014+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb24c140880 con 0x7fb24c0b1730 2024-03-25T19:23:59.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.039+0000 7fb241ffb640 1 -- 172.21.15.60:0/4101823333 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fb2480523b0 con 0x7fb24c0b1730 2024-03-25T19:23:59.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.039+0000 7fb241ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:23:59.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.039+0000 7fb241ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:23:59.039 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:23:59.039 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:23:59.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.041+0000 7fb253886640 10 librados: watch_flush enter 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.041+0000 7fb253886640 10 librados: watch_flush exit 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.041+0000 7fb253886640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.041+0000 7fb253886640 10 client.4378.objecter successfully canceled tick 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 >> v1:172.21.15.90:6800/1827638754 conn(0x7fb23006dfb0 legacy=0x7fb230070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 10 monclient: shutdown 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 >> v1:172.21.15.90:6789/0 conn(0x7fb24c0b1730 legacy=0x7fb24c1425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 shutdown_connections 2024-03-25T19:23:59.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 >> 172.21.15.60:0/4101823333 conn(0x7fb24c0617a0 msgr2=0x7fb24c13c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:23:59.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 shutdown_connections 2024-03-25T19:23:59.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 1 -- 172.21.15.60:0/4101823333 wait complete. 2024-03-25T19:23:59.043 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:23:59.042+0000 7fb253886640 1 librados: shutdown 2024-03-25T19:23:59.054 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:05.860 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.979+0000 7fd5b923e640 10 monclient: get_monmap_and_config 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.979+0000 7fd5b923e640 10 monclient: build_initial_monmap 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.979+0000 7fd5b923e640 10 monclient: monmap: 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:05.980149+0000 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:05.980149+0000 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:05.979 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:05.980 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:05.980 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.980+0000 7fd5b923e640 1 Processor -- start 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.980+0000 7fd5b923e640 1 -- start start 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.980+0000 7fd5b923e640 10 monclient: init 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b40b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b40b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40af940 con 0x7fd5b414be90 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40b0a70 con 0x7fd5b40b1730 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b40b0d50 con 0x7fd5b40b4d60 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:05.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.981+0000 7fd5b923e640 10 monclient(hunting): authenticate will time out at 6573.079127s 2024-03-25T19:24:05.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.991+0000 7fd5abfff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd5b40b1730 0x7fd5b40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:47778/0 (socket says 172.21.15.60:47778) 2024-03-25T19:24:05.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:05.991+0000 7fd5abfff640 1 -- 172.21.15.60:0/4248542289 learned_addr learned my addr 172.21.15.60:0/4248542289 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:06.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.004+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528751043 0 0) 0x7fd5b40b0a70 con 0x7fd5b40b1730 2024-03-25T19:24:06.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.004+0000 7fd5b259c640 10 monclient(hunting): _negotiate 2024-03-25T19:24:06.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.004+0000 7fd5b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:06.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.004+0000 7fd5b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:06.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.004+0000 7fd5b259c640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:24:06.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.004+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a0003520 con 0x7fd5b40b1730 2024-03-25T19:24:06.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.016+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991257833 0 0) 0x7fd5b40af940 con 0x7fd5b414be90 2024-03-25T19:24:06.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.016+0000 7fd5b259c640 10 monclient(hunting): _negotiate 2024-03-25T19:24:06.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.016+0000 7fd5b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:06.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.016+0000 7fd5b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:06.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.016+0000 7fd5b259c640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:24:06.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.016+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b40b0a70 con 0x7fd5b414be90 2024-03-25T19:24:06.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.018+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107475621 0 0) 0x7fd5b40b0d50 con 0x7fd5b40b4d60 2024-03-25T19:24:06.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.018+0000 7fd5b259c640 10 monclient(hunting): _negotiate 2024-03-25T19:24:06.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.018+0000 7fd5b259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:06.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.018+0000 7fd5b259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:06.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.018+0000 7fd5b259c640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:24:06.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.018+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b40af940 con 0x7fd5b40b4d60 2024-03-25T19:24:06.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974460690 0 0) 0x7fd5a0003520 con 0x7fd5b40b1730 2024-03-25T19:24:06.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 >> v1:172.21.15.60:6790/0 conn(0x7fd5b40b4d60 legacy=0x7fd5b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:06.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 >> v1:172.21.15.60:6789/0 conn(0x7fd5b414be90 legacy=0x7fd5b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:06.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 10 monclient: _finish_hunting 0 2024-03-25T19:24:06.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 1 monclient: found mon.noname-c 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b40b1030 con 0x7fd5b40b1730 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 10 monclient: _finish_auth 0 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 10 monclient: _check_auth_tickets 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd59c001690 con 0x7fd5b40b1730 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b259c640 10 monclient: dump: 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.022 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b923e640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:24:06.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.021+0000 7fd5b923e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd59c0017f0 con 0x7fd5b40b1730 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b259c640 1 -- 172.21.15.60:0/4248542289 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd59c001c00 con 0x7fd5b40b1730 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b259c640 10 monclient: dump: 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:06.050 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:06.051 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:06.051 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:06.051 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:06.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b923e640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:06.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.050+0000 7fd5b923e640 4 set_mon_vals no callback set 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.051+0000 7fd5b923e640 10 monclient: shutdown 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.051+0000 7fd5b923e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.051+0000 7fd5b923e640 1 -- 172.21.15.60:0/4248542289 >> v1:172.21.15.90:6789/0 conn(0x7fd5b40b1730 legacy=0x7fd5b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.051+0000 7fd5b923e640 1 -- 172.21.15.60:0/4248542289 shutdown_connections 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.051+0000 7fd5b923e640 1 -- 172.21.15.60:0/4248542289 >> 172.21.15.60:0/4248542289 conn(0x7fd5b40617a0 msgr2=0x7fd5b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.051+0000 7fd5b923e640 1 -- 172.21.15.60:0/4248542289 shutdown_connections 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.051+0000 7fd5b923e640 1 -- 172.21.15.60:0/4248542289 wait complete. 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 10 monclient: build_initial_monmap 2024-03-25T19:24:06.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 10 monclient: monmap: 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 1 librados: starting msgr at 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 1 librados: starting objecter 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 1 Processor -- start 2024-03-25T19:24:06.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 1 -- start start 2024-03-25T19:24:06.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 1 librados: setting wanted keys 2024-03-25T19:24:06.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 1 librados: calling monclient init 2024-03-25T19:24:06.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.052+0000 7fd5b923e640 10 monclient: init 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b40b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b40b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b41d83d0 con 0x7fd5b414be90 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b41d77a0 con 0x7fd5b40b1730 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b4138270 con 0x7fd5b40b4d60 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:06.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.053+0000 7fd5b923e640 10 monclient(hunting): authenticate will time out at 6573.151502s 2024-03-25T19:24:06.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.057+0000 7fd5b359e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd5b414be90 0x7fd5b4142d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52764/0 (socket says 172.21.15.60:52764) 2024-03-25T19:24:06.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.057+0000 7fd5b359e640 1 -- 172.21.15.60:0/1776850364 learned_addr learned my addr 172.21.15.60:0/1776850364 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:06.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.059+0000 7fd5ab7fe640 10 client.?.objecter ms_handle_connect 0x7fd5b40b4d60 2024-03-25T19:24:06.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.059+0000 7fd5ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:06.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.062+0000 7fd5ab7fe640 10 client.?.objecter ms_handle_connect 0x7fd5b40b1730 2024-03-25T19:24:06.061 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.062+0000 7fd5ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:06.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.063+0000 7fd5ab7fe640 10 client.?.objecter ms_handle_connect 0x7fd5b414be90 2024-03-25T19:24:06.062 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.063+0000 7fd5ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:06.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.066+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746515933 0 0) 0x7fd5b4138270 con 0x7fd5b40b4d60 2024-03-25T19:24:06.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.066+0000 7fd5ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:06.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.066+0000 7fd5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:06.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.066+0000 7fd5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:06.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.066+0000 7fd5ab7fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:24:06.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.066+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd594004eb0 con 0x7fd5b40b4d60 2024-03-25T19:24:06.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.068+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123351562 0 0) 0x7fd5b41d83d0 con 0x7fd5b414be90 2024-03-25T19:24:06.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.068+0000 7fd5ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:06.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.068+0000 7fd5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:06.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.068+0000 7fd5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:06.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.068+0000 7fd5ab7fe640 10 monclient(hunting): my global_id is 4395 2024-03-25T19:24:06.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.068+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b4138270 con 0x7fd5b414be90 2024-03-25T19:24:06.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.070+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061106130 0 0) 0x7fd5b41d77a0 con 0x7fd5b40b1730 2024-03-25T19:24:06.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.070+0000 7fd5ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:06.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.070+0000 7fd5ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:06.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.070+0000 7fd5ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:06.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.070+0000 7fd5ab7fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:24:06.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.070+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b41d83d0 con 0x7fd5b40b1730 2024-03-25T19:24:06.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.082+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005680377 0 0) 0x7fd5b4138270 con 0x7fd5b414be90 2024-03-25T19:24:06.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.082+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5b41d77a0 con 0x7fd5b414be90 2024-03-25T19:24:06.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.082+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd5ac001c10 con 0x7fd5b414be90 2024-03-25T19:24:06.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.082+0000 7fd5ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:06.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.082+0000 7fd5ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.082+0000 7fd5ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:06.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.082+0000 7fd5ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.086+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130125304 0 0) 0x7fd5b41d83d0 con 0x7fd5b40b1730 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.086+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5b4138270 con 0x7fd5b40b1730 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.086+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd5a4001830 con 0x7fd5b40b1730 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.086+0000 7fd5ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.086+0000 7fd5ab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.086+0000 7fd5ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:06.086 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:06.087 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.091+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897921531 0 0) 0x7fd594004eb0 con 0x7fd5b40b4d60 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.091+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5b41d83d0 con 0x7fd5b40b4d60 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.091+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd59c001b00 con 0x7fd5b40b4d60 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.091+0000 7fd5ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.091+0000 7fd5ab7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.091+0000 7fd5ab7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:06.091 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:06.092 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:06.092 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:06.092 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:06.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1052010383 0 0) 0x7fd5b41d77a0 con 0x7fd5b414be90 2024-03-25T19:24:06.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 >> v1:172.21.15.60:6790/0 conn(0x7fd5b40b4d60 legacy=0x7fd5b41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:06.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 >> v1:172.21.15.90:6789/0 conn(0x7fd5b40b1730 legacy=0x7fd5b41d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:06.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:06.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 1 monclient: found mon.a 2024-03-25T19:24:06.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:06.093 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4140d00 con 0x7fd5b414be90 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.092+0000 7fd5ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 5 monclient: authenticate success, global_id 4395 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 10 monclient: _renew_subs 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b4138a80 con 0x7fd5b414be90 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 10 monclient: _renew_subs 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b4143640 con 0x7fd5b414be90 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.093+0000 7fd5b923e640 1 librados: init done 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.094+0000 7fd5b923e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:06.094 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.094+0000 7fd5b923e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:06.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.094+0000 7fd5b923e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:06.095 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.094+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b4140ef0 con 0x7fd5b414be90 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.425+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5ac003970 con 0x7fd5b414be90 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.425+0000 7fd5ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.425+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd5ac002100 con 0x7fd5b414be90 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.425+0000 7fd5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.425+0000 7fd5ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.425+0000 7fd5ab7fe640 10 monclient: dump: 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:06.425 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:06.426 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:06.426 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:06.426 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:06.426 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:06.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.425+0000 7fd5b1d9b640 4 set_mon_vals no callback set 2024-03-25T19:24:06.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.426+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fd5ac003ca0 con 0x7fd5b414be90 2024-03-25T19:24:06.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.427+0000 7fd5b2d9d640 10 monclient: get_auth_request con 0x7fd59406e0b0 auth_method 0 2024-03-25T19:24:06.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.427+0000 7fd5ab7fe640 10 client.4395.objecter ms_handle_connect 0x7fd59406e0b0 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fd5ac054840 con 0x7fd5b414be90 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 10 client.4395.objecter ms_dispatch 0x7fd5b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5ac050350 con 0x7fd5b414be90 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.434+0000 7fd5ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:06.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.655+0000 7fd5b923e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:06.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.655+0000 7fd5b923e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:06.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.655+0000 7fd5b923e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:06.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.655+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5b40afbd0 con 0x7fd5b414be90 2024-03-25T19:24:06.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.682+0000 7fd5ab7fe640 1 -- 172.21.15.60:0/1776850364 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fd5ac052280 con 0x7fd5b414be90 2024-03-25T19:24:06.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.682+0000 7fd5ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:06.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.682+0000 7fd5ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:06.682 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:24:06.683 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:06.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 10 librados: watch_flush enter 2024-03-25T19:24:06.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 10 librados: watch_flush exit 2024-03-25T19:24:06.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 10 client.4395.objecter successfully canceled tick 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 >> v1:172.21.15.90:6800/1827638754 conn(0x7fd59406e0b0 legacy=0x7fd594070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 10 monclient: shutdown 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 >> v1:172.21.15.60:6789/0 conn(0x7fd5b414be90 legacy=0x7fd5b4142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 shutdown_connections 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 >> 172.21.15.60:0/1776850364 conn(0x7fd5b40617a0 msgr2=0x7fd5b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 shutdown_connections 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 1 -- 172.21.15.60:0/1776850364 wait complete. 2024-03-25T19:24:06.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:06.685+0000 7fd5b923e640 1 librados: shutdown 2024-03-25T19:24:06.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:13.505 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:13.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.604+0000 7f5989189640 10 monclient: get_monmap_and_config 2024-03-25T19:24:13.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.604+0000 7f5989189640 10 monclient: build_initial_monmap 2024-03-25T19:24:13.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.604+0000 7f5989189640 10 monclient: monmap: 2024-03-25T19:24:13.603 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:13.603 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:13.604777+0000 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:13.604777+0000 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:13.604 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:13.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.604+0000 7f5989189640 1 Processor -- start 2024-03-25T19:24:13.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.604+0000 7f5989189640 1 -- start start 2024-03-25T19:24:13.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient: init 2024-03-25T19:24:13.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:13.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient(hunting): picked mon.noname-b con 0x7f59840b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient(hunting): picked mon.noname-a con 0x7f59840b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient(hunting): picked mon.noname-c con 0x7f598414be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840af940 con 0x7f59840b1730 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840b0a70 con 0x7f598414be90 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840b0d50 con 0x7f59840b4d60 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:13.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.605+0000 7f5989189640 10 monclient(hunting): authenticate will time out at 6580.703556s 2024-03-25T19:24:13.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.612+0000 7f598359e640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f598414be90 0x7f598414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:52484/0 (socket says 172.21.15.60:52484) 2024-03-25T19:24:13.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.612+0000 7f598359e640 1 -- 172.21.15.60:0/912194896 learned_addr learned my addr 172.21.15.60:0/912194896 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:13.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.634+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182470736 0 0) 0x7f59840af940 con 0x7f59840b1730 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.634+0000 7f598259c640 10 monclient(hunting): _negotiate 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.634+0000 7f598259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.634+0000 7f598259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.634+0000 7f598259c640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.635+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5970003520 con 0x7f59840b1730 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.635+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027985474 0 0) 0x7f59840b0a70 con 0x7f598414be90 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.635+0000 7f598259c640 10 monclient(hunting): _negotiate 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.635+0000 7f598259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.635+0000 7f598259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.635+0000 7f598259c640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:24:13.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.635+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59840af940 con 0x7f598414be90 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2983097875 0 0) 0x7f5970003520 con 0x7f59840b1730 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 >> v1:172.21.15.60:6790/0 conn(0x7f59840b4d60 legacy=0x7f598413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 >> v1:172.21.15.90:6789/0 conn(0x7f598414be90 legacy=0x7f598414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 10 monclient: _finish_hunting 0 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 1 monclient: found mon.noname-a 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840b1030 con 0x7f59840b1730 2024-03-25T19:24:13.646 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 10 monclient: _finish_auth 0 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 10 monclient: _check_auth_tickets 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f596c001690 con 0x7f59840b1730 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f5989189640 5 monclient: authenticate success, global_id 4398 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f5989189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f598259c640 10 monclient: dump: 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:13.647 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:13.648 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:13.648 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:13.648 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:13.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.646+0000 7f5989189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f596c0017f0 con 0x7f59840b1730 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f598259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f598259c640 1 -- 172.21.15.60:0/912194896 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f596c001c00 con 0x7f59840b1730 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f5989189640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f5989189640 4 set_mon_vals no callback set 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f598259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f598259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.657+0000 7f598259c640 10 monclient: dump: 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:13.657 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:13.658 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.658+0000 7f5989189640 10 monclient: shutdown 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.658+0000 7f5989189640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.658+0000 7f5989189640 1 -- 172.21.15.60:0/912194896 >> v1:172.21.15.60:6789/0 conn(0x7f59840b1730 legacy=0x7f59840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.658+0000 7f5989189640 1 -- 172.21.15.60:0/912194896 shutdown_connections 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.658+0000 7f5989189640 1 -- 172.21.15.60:0/912194896 >> 172.21.15.60:0/912194896 conn(0x7f59840617a0 msgr2=0x7f5984061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.658+0000 7f5989189640 1 -- 172.21.15.60:0/912194896 shutdown_connections 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.658+0000 7f5989189640 1 -- 172.21.15.60:0/912194896 wait complete. 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 10 monclient: build_initial_monmap 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 10 monclient: monmap: 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:13.660 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 1 librados: starting msgr at 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 1 librados: starting objecter 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 1 Processor -- start 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 1 -- start start 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 1 librados: setting wanted keys 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 1 librados: calling monclient init 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.659+0000 7f5989189640 10 monclient: init 2024-03-25T19:24:13.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 10 monclient(hunting): picked mon.noname-a con 0x7f59840b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 10 monclient(hunting): picked mon.noname-c con 0x7f59840b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 10 monclient(hunting): picked mon.noname-b con 0x7f598414be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59841451e0 con 0x7f59840b1730 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59841411f0 con 0x7f598414be90 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5984141380 con 0x7f59840b4d60 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:13.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.660+0000 7f5989189640 10 monclient(hunting): authenticate will time out at 6580.758462s 2024-03-25T19:24:13.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.671+0000 7f597bfff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f59840b4d60 0x7f59841427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:48784/0 (socket says 172.21.15.60:48784) 2024-03-25T19:24:13.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.671+0000 7f597bfff640 1 -- 172.21.15.60:0/2715836005 learned_addr learned my addr 172.21.15.60:0/2715836005 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:13.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.672+0000 7f597b7fe640 10 client.?.objecter ms_handle_connect 0x7f59840b4d60 2024-03-25T19:24:13.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.672+0000 7f597b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:13.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.674+0000 7f597b7fe640 10 client.?.objecter ms_handle_connect 0x7f59840b1730 2024-03-25T19:24:13.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.674+0000 7f597b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:13.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.675+0000 7f597b7fe640 10 client.?.objecter ms_handle_connect 0x7f598414be90 2024-03-25T19:24:13.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.675+0000 7f597b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:13.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.681+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555074033 0 0) 0x7f59841451e0 con 0x7f59840b1730 2024-03-25T19:24:13.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.681+0000 7f597b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:13.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.681+0000 7f597b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:13.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.681+0000 7f597b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:13.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.681+0000 7f597b7fe640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:24:13.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.681+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5964004eb0 con 0x7f59840b1730 2024-03-25T19:24:13.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.687+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906575153 0 0) 0x7f59841411f0 con 0x7f598414be90 2024-03-25T19:24:13.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.687+0000 7f597b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:13.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.687+0000 7f597b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:13.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.687+0000 7f597b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:13.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.687+0000 7f597b7fe640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:24:13.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.687+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59841451e0 con 0x7f598414be90 2024-03-25T19:24:13.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.690+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453762462 0 0) 0x7f5984141380 con 0x7f59840b4d60 2024-03-25T19:24:13.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.691+0000 7f597b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:13.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.691+0000 7f597b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:13.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.691+0000 7f597b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:13.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.691+0000 7f597b7fe640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:24:13.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.691+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59841411f0 con 0x7f59840b4d60 2024-03-25T19:24:13.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.691+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669326186 0 0) 0x7f5964004eb0 con 0x7f59840b1730 2024-03-25T19:24:13.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.691+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5984141380 con 0x7f59840b1730 2024-03-25T19:24:13.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.693+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5974001980 con 0x7f59840b1730 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.693+0000 7f597b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.693+0000 7f597b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.693+0000 7f597b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:13.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.693+0000 7f597b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:13.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.703+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067068215 0 0) 0x7f59841451e0 con 0x7f598414be90 2024-03-25T19:24:13.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.703+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5964004eb0 con 0x7f598414be90 2024-03-25T19:24:13.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.703+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f597c001b90 con 0x7f598414be90 2024-03-25T19:24:13.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.703+0000 7f597b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:13.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.703+0000 7f597b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.703+0000 7f597b7fe640 10 monclient(hunting): dump: 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:13.703 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2802065297 0 0) 0x7f5984141380 con 0x7f59840b1730 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 >> v1:172.21.15.60:6790/0 conn(0x7f59840b4d60 legacy=0x7f59841427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 >> v1:172.21.15.90:6789/0 conn(0x7f598414be90 legacy=0x7f5984142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 1 monclient: found mon.a 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5984137eb0 con 0x7f59840b1730 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f597b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 10 monclient: _renew_subs 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:13.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59841380a0 con 0x7f59840b1730 2024-03-25T19:24:13.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:13.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 10 monclient: _renew_subs 2024-03-25T19:24:13.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:13.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59841e3710 con 0x7f59840b1730 2024-03-25T19:24:13.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.711+0000 7f5989189640 1 librados: init done 2024-03-25T19:24:13.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.712+0000 7f5989189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:13.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.712+0000 7f5989189640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:13.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.712+0000 7f5989189640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:13.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:13.712+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59841453d0 con 0x7f59840b1730 2024-03-25T19:24:14.047 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.048+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59740035f0 con 0x7f59840b1730 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.048+0000 7f597b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.048+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5974003900 con 0x7f59840b1730 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.048+0000 7f597b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.048+0000 7f597b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.048+0000 7f597b7fe640 10 monclient: dump: 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:14.048 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:14.049 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:14.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.048+0000 7f5981d9b640 4 set_mon_vals no callback set 2024-03-25T19:24:14.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.049+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f5974019950 con 0x7f59840b1730 2024-03-25T19:24:14.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.049+0000 7f597bfff640 10 monclient: get_auth_request con 0x7f596406df60 auth_method 0 2024-03-25T19:24:14.049 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.050+0000 7f597b7fe640 10 client.4401.objecter ms_handle_connect 0x7f596406df60 2024-03-25T19:24:14.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f59740536b0 con 0x7f59840b1730 2024-03-25T19:24:14.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 10 client.4401.objecter ms_dispatch 0x7f5984000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:14.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:14.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:14.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T19:24:14.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f597404eff0 con 0x7f59840b1730 2024-03-25T19:24:14.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:14.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.055+0000 7f597b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:14.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.315+0000 7f5989189640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:14.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.315+0000 7f5989189640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:14.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.315+0000 7f5989189640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:14.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.315+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59840627b0 con 0x7f59840b1730 2024-03-25T19:24:14.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.409+0000 7f597b7fe640 1 -- 172.21.15.60:0/2715836005 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f59740510f0 con 0x7f59840b1730 2024-03-25T19:24:14.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.409+0000 7f597b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:14.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.409+0000 7f597b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:14.409 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:24:14.409 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:14.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 10 librados: watch_flush enter 2024-03-25T19:24:14.411 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 10 librados: watch_flush exit 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 10 client.4401.objecter successfully canceled tick 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 >> v1:172.21.15.90:6800/1827638754 conn(0x7f596406df60 legacy=0x7f5964070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 10 monclient: shutdown 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 >> v1:172.21.15.60:6789/0 conn(0x7f59840b1730 legacy=0x7f5984140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.411+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 shutdown_connections 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.412+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 >> 172.21.15.60:0/2715836005 conn(0x7f59840617a0 msgr2=0x7f598413c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.412+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 shutdown_connections 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.412+0000 7f5989189640 1 -- 172.21.15.60:0/2715836005 wait complete. 2024-03-25T19:24:14.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:14.412+0000 7f5989189640 1 librados: shutdown 2024-03-25T19:24:14.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:21.227 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:21.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.343+0000 7fd1b52d4640 10 monclient: get_monmap_and_config 2024-03-25T19:24:21.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.343+0000 7fd1b52d4640 10 monclient: build_initial_monmap 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.343+0000 7fd1b52d4640 10 monclient: monmap: 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:21.344115+0000 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:21.344115+0000 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:21.346 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:21.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.344+0000 7fd1b52d4640 1 Processor -- start 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.344+0000 7fd1b52d4640 1 -- start start 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.344+0000 7fd1b52d4640 10 monclient: init 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.344+0000 7fd1b52d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.344+0000 7fd1b52d4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.344+0000 7fd1b52d4640 10 monclient(hunting): picked mon.noname-a con 0x7fd1b006ab80 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.344+0000 7fd1b52d4640 10 monclient(hunting): picked mon.noname-b con 0x7fd1b006af60 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.345+0000 7fd1b52d4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1b013f990 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.345+0000 7fd1b52d4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b00afa60 con 0x7fd1b006ab80 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.345+0000 7fd1b52d4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b00afce0 con 0x7fd1b013f990 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.345+0000 7fd1b52d4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b00b1150 con 0x7fd1b006af60 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.345+0000 7fd1b52d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:21.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.345+0000 7fd1b52d4640 10 monclient(hunting): authenticate will time out at 6588.442932s 2024-03-25T19:24:21.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.355+0000 7fd1af7fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd1b013f990 0x7fd1b014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:33382/0 (socket says 172.21.15.60:33382) 2024-03-25T19:24:21.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.355+0000 7fd1af7fe640 1 -- 172.21.15.60:0/3151591306 learned_addr learned my addr 172.21.15.60:0/3151591306 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:21.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.369+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572343018 0 0) 0x7fd1b00b1150 con 0x7fd1b006af60 2024-03-25T19:24:21.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.369+0000 7fd1ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:21.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.369+0000 7fd1ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:21.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.369+0000 7fd1ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:21.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.369+0000 7fd1ae7fc640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:24:21.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.369+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19c003520 con 0x7fd1b006af60 2024-03-25T19:24:21.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.371+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937315709 0 0) 0x7fd1b00afa60 con 0x7fd1b006ab80 2024-03-25T19:24:21.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.371+0000 7fd1ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:21.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.371+0000 7fd1ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:21.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.371+0000 7fd1ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:21.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.371+0000 7fd1ae7fc640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:24:21.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.371+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b00b1150 con 0x7fd1b006ab80 2024-03-25T19:24:21.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.377+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3078326473 0 0) 0x7fd1b00b1150 con 0x7fd1b006ab80 2024-03-25T19:24:21.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.377+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 >> v1:172.21.15.60:6790/0 conn(0x7fd1b006af60 legacy=0x7fd1b01458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.377+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 >> v1:172.21.15.90:6789/0 conn(0x7fd1b013f990 legacy=0x7fd1b014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 1 monclient: found mon.noname-a 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b00aff20 con 0x7fd1b006ab80 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd1a0001670 con 0x7fd1b006ab80 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1ae7fc640 10 monclient: dump: 2024-03-25T19:24:21.378 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1b52d4640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:24:21.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.378+0000 7fd1b52d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:21.403 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1a00017d0 con 0x7fd1b006ab80 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1ae7fc640 1 -- 172.21.15.60:0/3151591306 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd1a0001ba0 con 0x7fd1b006ab80 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1b52d4640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1b52d4640 4 set_mon_vals no callback set 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.404+0000 7fd1ae7fc640 10 monclient: dump: 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:21.404 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:21.405 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:21.405 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:21.405 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:21.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.405+0000 7fd1b52d4640 10 monclient: shutdown 2024-03-25T19:24:21.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.405+0000 7fd1b52d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:21.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.405+0000 7fd1b52d4640 1 -- 172.21.15.60:0/3151591306 >> v1:172.21.15.60:6789/0 conn(0x7fd1b006ab80 legacy=0x7fd1b00724e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:21.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.405+0000 7fd1b52d4640 1 -- 172.21.15.60:0/3151591306 shutdown_connections 2024-03-25T19:24:21.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.405+0000 7fd1b52d4640 1 -- 172.21.15.60:0/3151591306 >> 172.21.15.60:0/3151591306 conn(0x7fd1b006c830 msgr2=0x7fd1b00b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:21.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.405+0000 7fd1b52d4640 1 -- 172.21.15.60:0/3151591306 shutdown_connections 2024-03-25T19:24:21.407 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.405+0000 7fd1b52d4640 1 -- 172.21.15.60:0/3151591306 wait complete. 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 10 monclient: build_initial_monmap 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 10 monclient: monmap: 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 1 librados: starting msgr at 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 1 librados: starting objecter 2024-03-25T19:24:21.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 1 Processor -- start 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 1 -- start start 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 1 librados: setting wanted keys 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 1 librados: calling monclient init 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.406+0000 7fd1b52d4640 10 monclient: init 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1b006ab80 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 10 monclient(hunting): picked mon.noname-a con 0x7fd1b013f990 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 10 monclient(hunting): picked mon.noname-b con 0x7fd1b014e450 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b01db850 con 0x7fd1b013f990 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b01db2e0 con 0x7fd1b014e450 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b01db470 con 0x7fd1b006ab80 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:21.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.407+0000 7fd1b52d4640 10 monclient(hunting): authenticate will time out at 6588.505866s 2024-03-25T19:24:21.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.420+0000 7fd1aeffd640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fd1b006ab80 0x7fd1b01dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:46534/0 (socket says 172.21.15.60:46534) 2024-03-25T19:24:21.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.420+0000 7fd1aeffd640 1 -- 172.21.15.60:0/1949825607 learned_addr learned my addr 172.21.15.60:0/1949825607 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:21.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.428+0000 7fd1a77fe640 10 client.?.objecter ms_handle_connect 0x7fd1b014e450 2024-03-25T19:24:21.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.428+0000 7fd1a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:21.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.434+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139392388 0 0) 0x7fd1b01db2e0 con 0x7fd1b014e450 2024-03-25T19:24:21.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.434+0000 7fd1a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:21.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.434+0000 7fd1a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:21.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.434+0000 7fd1a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:21.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.434+0000 7fd1a77fe640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:24:21.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.434+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190004eb0 con 0x7fd1b014e450 2024-03-25T19:24:21.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.435+0000 7fd1a77fe640 10 client.?.objecter ms_handle_connect 0x7fd1b013f990 2024-03-25T19:24:21.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.435+0000 7fd1a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:21.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.435+0000 7fd1a77fe640 10 client.?.objecter ms_handle_connect 0x7fd1b006ab80 2024-03-25T19:24:21.435 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.435+0000 7fd1a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:21.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.449+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045974947 0 0) 0x7fd1b01db470 con 0x7fd1b006ab80 2024-03-25T19:24:21.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.449+0000 7fd1a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:21.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.449+0000 7fd1a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:21.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.449+0000 7fd1a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:21.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.449+0000 7fd1a77fe640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:24:21.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.449+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b01db2e0 con 0x7fd1b006ab80 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.452+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172938434 0 0) 0x7fd190004eb0 con 0x7fd1b014e450 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.452+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1b01db470 con 0x7fd1b014e450 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.452+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd1a8001b00 con 0x7fd1b014e450 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.452+0000 7fd1a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.452+0000 7fd1a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.452+0000 7fd1a77fe640 10 monclient(hunting): dump: 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:21.452 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:21.453 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:21.453 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.452+0000 7fd1a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:21.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.455+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568521911 0 0) 0x7fd1b01db850 con 0x7fd1b013f990 2024-03-25T19:24:21.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.455+0000 7fd1a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:24:21.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.455+0000 7fd1a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:21.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.455+0000 7fd1a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:21.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.455+0000 7fd1a77fe640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:24:21.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.455+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190004eb0 con 0x7fd1b013f990 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.461+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051222182 0 0) 0x7fd1b01db2e0 con 0x7fd1b006ab80 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.461+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1b01db850 con 0x7fd1b006ab80 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.461+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd1a0001ba0 con 0x7fd1b006ab80 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.461+0000 7fd1a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.461+0000 7fd1a77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.461+0000 7fd1a77fe640 10 monclient(hunting): dump: 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.461 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:21.462 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:21.462 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:21.462 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:21.462 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:21.462 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:21.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2479075430 0 0) 0x7fd1b01db470 con 0x7fd1b014e450 2024-03-25T19:24:21.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 >> v1:172.21.15.60:6790/0 conn(0x7fd1b006ab80 legacy=0x7fd1b01dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 >> v1:172.21.15.60:6789/0 conn(0x7fd1b013f990 legacy=0x7fd1b0139df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 1 monclient: found mon.b 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b01db600 con 0x7fd1b014e450 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 10 monclient: _finish_auth 0 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 5 monclient: authenticate success, global_id 4396 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 10 monclient: _renew_subs 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b013a500 con 0x7fd1b014e450 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:21.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 10 monclient: _renew_subs 2024-03-25T19:24:21.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:21.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.464+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b013dc60 con 0x7fd1b014e450 2024-03-25T19:24:21.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.465+0000 7fd1b52d4640 1 librados: init done 2024-03-25T19:24:21.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.466+0000 7fd1b52d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:21.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.466+0000 7fd1b52d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:21.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.466+0000 7fd1b52d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:21.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.466+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b013f6b0 con 0x7fd1b014e450 2024-03-25T19:24:21.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.774+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1a8001f70 con 0x7fd1b014e450 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.774+0000 7fd1a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.774+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd1a8004d00 con 0x7fd1b014e450 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.774+0000 7fd1a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.774+0000 7fd1a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.774+0000 7fd1a77fe640 10 monclient: dump: 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:21.774 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:21.775 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:21.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.774+0000 7fd1adffb640 4 set_mon_vals no callback set 2024-03-25T19:24:21.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.775+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fd1a801acd0 con 0x7fd1b014e450 2024-03-25T19:24:21.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.775+0000 7fd1aeffd640 10 monclient: get_auth_request con 0x7fd19006df70 auth_method 0 2024-03-25T19:24:21.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.775+0000 7fd1a77fe640 10 client.4396.objecter ms_handle_connect 0x7fd19006df70 2024-03-25T19:24:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.780+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fd1a8054a10 con 0x7fd1b014e450 2024-03-25T19:24:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.780+0000 7fd1a77fe640 10 client.4396.objecter ms_dispatch 0x7fd1b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.780+0000 7fd1a77fe640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.780+0000 7fd1a77fe640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.780+0000 7fd1a77fe640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T19:24:21.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.781+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1a8050350 con 0x7fd1b014e450 2024-03-25T19:24:21.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.781+0000 7fd1a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:21.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:21.781+0000 7fd1a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:22.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.021+0000 7fd1b52d4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:22.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.021+0000 7fd1b52d4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:22.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.021+0000 7fd1b52d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:22.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.021+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1b00627b0 con 0x7fd1b014e450 2024-03-25T19:24:22.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.048+0000 7fd1a77fe640 1 -- 172.21.15.60:0/1949825607 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fd1a8052450 con 0x7fd1b014e450 2024-03-25T19:24:22.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.048+0000 7fd1a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:22.048 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.048+0000 7fd1a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:22.048 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:24:22.048 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 10 librados: watch_flush enter 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 10 librados: watch_flush exit 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 10 client.4396.objecter successfully canceled tick 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 >> v1:172.21.15.90:6800/1827638754 conn(0x7fd19006df70 legacy=0x7fd190070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 10 monclient: shutdown 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 >> v1:172.21.15.90:6789/0 conn(0x7fd1b014e450 legacy=0x7fd1b013d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 shutdown_connections 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 >> 172.21.15.60:0/1949825607 conn(0x7fd1b006c830 msgr2=0x7fd1b00b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:22.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 shutdown_connections 2024-03-25T19:24:22.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 1 -- 172.21.15.60:0/1949825607 wait complete. 2024-03-25T19:24:22.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:22.051+0000 7fd1b52d4640 1 librados: shutdown 2024-03-25T19:24:22.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:28.868 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.981+0000 7f8c962a8640 10 monclient: get_monmap_and_config 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.981+0000 7f8c962a8640 10 monclient: build_initial_monmap 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.981+0000 7f8c962a8640 10 monclient: monmap: 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:28.982228+0000 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:28.982228+0000 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:28.981 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:28.982 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 1 Processor -- start 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 1 -- start start 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient: init 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8c900b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8c900b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8c9014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c900af940 con 0x7f8c900b1730 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c900b0a70 con 0x7f8c900b4d60 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c900b0d50 con 0x7f8c9014be90 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:28.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:28.982+0000 7f8c962a8640 10 monclient(hunting): authenticate will time out at 6596.080852s 2024-03-25T19:24:29.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.000+0000 7f8c94845640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f8c9014be90 0x7f8c9014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:48666/0 (socket says 172.21.15.60:48666) 2024-03-25T19:24:29.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.000+0000 7f8c94845640 1 -- 172.21.15.60:0/2962281350 learned_addr learned my addr 172.21.15.60:0/2962281350 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:29.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.010+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495449746 0 0) 0x7f8c900b0d50 con 0x7f8c9014be90 2024-03-25T19:24:29.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.010+0000 7f8c8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:29.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.010+0000 7f8c8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:29.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.010+0000 7f8c8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:29.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.010+0000 7f8c8effd640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:24:29.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.010+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c78003520 con 0x7f8c9014be90 2024-03-25T19:24:29.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.013+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675530086 0 0) 0x7f8c900b0a70 con 0x7f8c900b4d60 2024-03-25T19:24:29.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.013+0000 7f8c8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:29.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.013+0000 7f8c8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:29.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.013+0000 7f8c8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:29.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.013+0000 7f8c8effd640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:24:29.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.013+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c900b0d50 con 0x7f8c900b4d60 2024-03-25T19:24:29.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.026+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366579143 0 0) 0x7f8c900af940 con 0x7f8c900b1730 2024-03-25T19:24:29.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.026+0000 7f8c8effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:29.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.026+0000 7f8c8effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:29.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.026+0000 7f8c8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:29.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.026+0000 7f8c8effd640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:24:29.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.026+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c900b0a70 con 0x7f8c900b1730 2024-03-25T19:24:29.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3286242080 0 0) 0x7f8c78003520 con 0x7f8c9014be90 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 >> v1:172.21.15.90:6789/0 conn(0x7f8c900b4d60 legacy=0x7f8c9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 >> v1:172.21.15.60:6789/0 conn(0x7f8c900b1730 legacy=0x7f8c900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 1 monclient: found mon.noname-b 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c900b1030 con 0x7f8c9014be90 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 10 monclient: _finish_auth 0 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8c88001a10 con 0x7f8c9014be90 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c962a8640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c962a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:29.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c8effd640 10 monclient: dump: 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:29.030 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:29.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.028+0000 7f8c962a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c88001b70 con 0x7f8c9014be90 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c8effd640 1 -- 172.21.15.60:0/2962281350 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8c88001f80 con 0x7f8c9014be90 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c8effd640 10 monclient: dump: 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:29.066 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c962a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:29.067 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.066+0000 7f8c962a8640 4 set_mon_vals no callback set 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.067+0000 7f8c962a8640 10 monclient: shutdown 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.067+0000 7f8c962a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.067+0000 7f8c962a8640 1 -- 172.21.15.60:0/2962281350 >> v1:172.21.15.60:6790/0 conn(0x7f8c9014be90 legacy=0x7f8c9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.067+0000 7f8c962a8640 1 -- 172.21.15.60:0/2962281350 shutdown_connections 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.067+0000 7f8c962a8640 1 -- 172.21.15.60:0/2962281350 >> 172.21.15.60:0/2962281350 conn(0x7f8c900617a0 msgr2=0x7f8c90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.067+0000 7f8c962a8640 1 -- 172.21.15.60:0/2962281350 shutdown_connections 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.067+0000 7f8c962a8640 1 -- 172.21.15.60:0/2962281350 wait complete. 2024-03-25T19:24:29.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.068+0000 7f8c962a8640 10 monclient: build_initial_monmap 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.068+0000 7f8c962a8640 10 monclient: monmap: 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.068+0000 7f8c962a8640 1 librados: starting msgr at 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.068+0000 7f8c962a8640 1 librados: starting objecter 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.068+0000 7f8c962a8640 1 Processor -- start 2024-03-25T19:24:29.070 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 1 -- start start 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 1 librados: setting wanted keys 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 1 librados: calling monclient init 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient: init 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8c900b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8c900b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8c9014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c90141a50 con 0x7f8c900b1730 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c90135810 con 0x7f8c900b4d60 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c90141cc0 con 0x7f8c9014be90 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:29.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.069+0000 7f8c962a8640 10 monclient(hunting): authenticate will time out at 6596.167611s 2024-03-25T19:24:29.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.076+0000 7f8c8f7fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8c900b4d60 0x7f8c901349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:42920/0 (socket says 172.21.15.60:42920) 2024-03-25T19:24:29.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.076+0000 7f8c8f7fe640 1 -- 172.21.15.60:0/1261495147 learned_addr learned my addr 172.21.15.60:0/1261495147 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:29.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.077+0000 7f8c8cff9640 10 client.?.objecter ms_handle_connect 0x7f8c900b4d60 2024-03-25T19:24:29.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.077+0000 7f8c8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:29.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.086+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764627077 0 0) 0x7f8c90135810 con 0x7f8c900b4d60 2024-03-25T19:24:29.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.086+0000 7f8c8cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:29.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.086+0000 7f8c8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:29.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.086+0000 7f8c8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:29.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.086+0000 7f8c8cff9640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:24:29.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.086+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c7c004eb0 con 0x7f8c900b4d60 2024-03-25T19:24:29.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.088+0000 7f8c8cff9640 10 client.?.objecter ms_handle_connect 0x7f8c9014be90 2024-03-25T19:24:29.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.088+0000 7f8c8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:29.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.091+0000 7f8c8cff9640 10 client.?.objecter ms_handle_connect 0x7f8c900b1730 2024-03-25T19:24:29.090 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.091+0000 7f8c8cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.103+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369577849 0 0) 0x7f8c7c004eb0 con 0x7f8c900b4d60 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.103+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c90135810 con 0x7f8c900b4d60 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.103+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8c84001a10 con 0x7f8c900b4d60 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.103+0000 7f8c8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.103+0000 7f8c8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.103+0000 7f8c8cff9640 10 monclient(hunting): dump: 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:29.103 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:29.104 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:29.104 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:29.104 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:29.104 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.103+0000 7f8c8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:29.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.113+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537609569 0 0) 0x7f8c90141cc0 con 0x7f8c9014be90 2024-03-25T19:24:29.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.113+0000 7f8c8cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:29.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.113+0000 7f8c8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:29.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.113+0000 7f8c8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:29.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.113+0000 7f8c8cff9640 10 monclient(hunting): my global_id is 4397 2024-03-25T19:24:29.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.113+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c7c004eb0 con 0x7f8c9014be90 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 961423377 0 0) 0x7f8c90135810 con 0x7f8c900b4d60 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 >> v1:172.21.15.60:6790/0 conn(0x7f8c9014be90 legacy=0x7f8c90135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 >> v1:172.21.15.60:6789/0 conn(0x7f8c900b1730 legacy=0x7f8c901dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 1 monclient: found mon.b 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c90140810 con 0x7f8c900b4d60 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 10 monclient: _finish_auth 0 2024-03-25T19:24:29.119 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 10 monclient: _renew_subs 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c901377a0 con 0x7f8c900b4d60 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 10 monclient: _renew_subs 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c90140560 con 0x7f8c900b4d60 2024-03-25T19:24:29.120 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.119+0000 7f8c962a8640 1 librados: init done 2024-03-25T19:24:29.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.120+0000 7f8c962a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:29.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.120+0000 7f8c962a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:29.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.120+0000 7f8c962a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:29.121 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.120+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c900627b0 con 0x7f8c900b4d60 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.337+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c84001cb0 con 0x7f8c900b4d60 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.337+0000 7f8c8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.337+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8c84003a60 con 0x7f8c900b4d60 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.337+0000 7f8c8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.337+0000 7f8c8cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.337+0000 7f8c8cff9640 10 monclient: dump: 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:29.337 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:29.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.337+0000 7f8c8e7fc640 4 set_mon_vals no callback set 2024-03-25T19:24:29.404 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.404+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f8c84003bc0 con 0x7f8c900b4d60 2024-03-25T19:24:29.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.405+0000 7f8c8ffff640 10 monclient: get_auth_request con 0x7f8c7c06dd60 auth_method 0 2024-03-25T19:24:29.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.406+0000 7f8c8cff9640 10 client.4402.objecter ms_handle_connect 0x7f8c7c06dd60 2024-03-25T19:24:29.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.411+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f8c840548b0 con 0x7f8c900b4d60 2024-03-25T19:24:29.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.411+0000 7f8c8cff9640 10 client.4402.objecter ms_dispatch 0x7f8c90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:29.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.411+0000 7f8c8cff9640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:29.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.411+0000 7f8c8cff9640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:29.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.411+0000 7f8c8cff9640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:24:29.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.414+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c840501f0 con 0x7f8c900b4d60 2024-03-25T19:24:29.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.414+0000 7f8c8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:29.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.414+0000 7f8c8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:29.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.668+0000 7f8c962a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:29.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.668+0000 7f8c962a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:29.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.668+0000 7f8c962a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:29.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.668+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c901409a0 con 0x7f8c900b4d60 2024-03-25T19:24:29.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.784+0000 7f8c8cff9640 1 -- 172.21.15.60:0/1261495147 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f8c840522f0 con 0x7f8c900b4d60 2024-03-25T19:24:29.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.784+0000 7f8c8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:29.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.784+0000 7f8c8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:29.784 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:24:29.784 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:29.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 10 librados: watch_flush enter 2024-03-25T19:24:29.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 10 librados: watch_flush exit 2024-03-25T19:24:29.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 >> v1:172.21.15.90:6800/1827638754 conn(0x7f8c7c06dd60 legacy=0x7f8c7c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 10 monclient: shutdown 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.786+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 >> v1:172.21.15.90:6789/0 conn(0x7f8c900b4d60 legacy=0x7f8c901349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.787+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 shutdown_connections 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.787+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 >> 172.21.15.60:0/1261495147 conn(0x7f8c900617a0 msgr2=0x7f8c9013c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.787+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 shutdown_connections 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.787+0000 7f8c962a8640 1 -- 172.21.15.60:0/1261495147 wait complete. 2024-03-25T19:24:29.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:29.787+0000 7f8c962a8640 1 librados: shutdown 2024-03-25T19:24:29.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:34.446 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:04:35.158 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.454 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:04:35.166 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.483 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:04:35.194 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.510 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:04:35.221 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.521 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:04:35.232 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.527 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:35.181 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.534 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:35.188 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.550 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:04:35.261 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:34.559 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:35.213 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:36.605 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:36.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.709+0000 7fe5b742a640 10 monclient: get_monmap_and_config 2024-03-25T19:24:36.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.709+0000 7fe5b742a640 10 monclient: build_initial_monmap 2024-03-25T19:24:36.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.709+0000 7fe5b742a640 10 monclient: monmap: 2024-03-25T19:24:36.708 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:36.708 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:36.708 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:36.709776+0000 2024-03-25T19:24:36.708 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:36.709776+0000 2024-03-25T19:24:36.709 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:36.709 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:36.709 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:36.709 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:36.709 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:36.709 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:36.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.709+0000 7fe5b742a640 1 Processor -- start 2024-03-25T19:24:36.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.709+0000 7fe5b742a640 1 -- start start 2024-03-25T19:24:36.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.709+0000 7fe5b742a640 10 monclient: init 2024-03-25T19:24:36.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:36.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b00b10d0 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b00b4360 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b00a9470 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00a9ab0 con 0x7fe5b00b4360 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00b2400 con 0x7fe5b00b10d0 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00aa2a0 con 0x7fe5b00a9470 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.710+0000 7fe5b742a640 10 monclient(hunting): authenticate will time out at 6603.808415s 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.711+0000 7fe5b59c7640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fe5b00a9470 0x7fe5b014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:46782/0 (socket says 172.21.15.60:46782) 2024-03-25T19:24:36.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.711+0000 7fe5b59c7640 1 -- 172.21.15.60:0/144954226 learned_addr learned my addr 172.21.15.60:0/144954226 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:36.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.734+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016397636 0 0) 0x7fe5b00a9ab0 con 0x7fe5b00b4360 2024-03-25T19:24:36.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.734+0000 7fe5a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:24:36.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.734+0000 7fe5a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:36.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.734+0000 7fe5a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:36.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.734+0000 7fe5a7fff640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:24:36.733 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.734+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe59c003520 con 0x7fe5b00b4360 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 106546197 0 0) 0x7fe59c003520 con 0x7fe5b00b4360 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 >> v1:172.21.15.60:6790/0 conn(0x7fe5b00a9470 legacy=0x7fe5b014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 >> v1:172.21.15.90:6789/0 conn(0x7fe5b00b10d0 legacy=0x7fe5b00b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 1 monclient: found mon.noname-a 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b00b14b0 con 0x7fe5b00b4360 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 10 monclient: _finish_auth 0 2024-03-25T19:24:36.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5a0001690 con 0x7fe5b00b4360 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5b742a640 5 monclient: authenticate success, global_id 4416 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5b742a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5a7fff640 10 monclient: dump: 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:36.767 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:36.768 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:36.768 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:36.768 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:36.768 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:36.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.766+0000 7fe5b742a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5a00017f0 con 0x7fe5b00b4360 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5a7fff640 1 -- 172.21.15.60:0/144954226 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5a0001c00 con 0x7fe5b00b4360 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5b742a640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5b742a640 4 set_mon_vals no callback set 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.781+0000 7fe5a7fff640 10 monclient: dump: 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:36.781 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:36.782 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:36.782 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:36.782 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:36.782 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:36.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.782+0000 7fe5b742a640 10 monclient: shutdown 2024-03-25T19:24:36.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.782+0000 7fe5b742a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:36.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.782+0000 7fe5b742a640 1 -- 172.21.15.60:0/144954226 >> v1:172.21.15.60:6789/0 conn(0x7fe5b00b4360 legacy=0x7fe5b00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:36.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.782+0000 7fe5b742a640 1 -- 172.21.15.60:0/144954226 shutdown_connections 2024-03-25T19:24:36.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.782+0000 7fe5b742a640 1 -- 172.21.15.60:0/144954226 >> 172.21.15.60:0/144954226 conn(0x7fe5b00617a0 msgr2=0x7fe5b0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:36.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.782+0000 7fe5b742a640 1 -- 172.21.15.60:0/144954226 shutdown_connections 2024-03-25T19:24:36.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.782+0000 7fe5b742a640 1 -- 172.21.15.60:0/144954226 wait complete. 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.783+0000 7fe5b742a640 10 monclient: build_initial_monmap 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.783+0000 7fe5b742a640 10 monclient: monmap: 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.783+0000 7fe5b742a640 1 librados: starting msgr at 2024-03-25T19:24:36.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.783+0000 7fe5b742a640 1 librados: starting objecter 2024-03-25T19:24:36.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.783+0000 7fe5b742a640 1 Processor -- start 2024-03-25T19:24:36.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 1 -- start start 2024-03-25T19:24:36.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 1 librados: setting wanted keys 2024-03-25T19:24:36.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 1 librados: calling monclient init 2024-03-25T19:24:36.785 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient: init 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient(hunting): picked mon.noname-b con 0x7fe5b00a9470 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient(hunting): picked mon.noname-c con 0x7fe5b00b10d0 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient(hunting): picked mon.noname-a con 0x7fe5b00b4360 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b01398c0 con 0x7fe5b00b4360 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00b0990 con 0x7fe5b00a9470 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b013e7c0 con 0x7fe5b00b10d0 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:36.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.784+0000 7fe5b742a640 10 monclient(hunting): authenticate will time out at 6603.882786s 2024-03-25T19:24:36.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.794+0000 7fe5b49c5640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fe5b00b10d0 0x7fe5b0138aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:46822/0 (socket says 172.21.15.60:46822) 2024-03-25T19:24:36.793 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.794+0000 7fe5b49c5640 1 -- 172.21.15.60:0/180836254 learned_addr learned my addr 172.21.15.60:0/180836254 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:36.795 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.795+0000 7fe5a5ffb640 10 client.?.objecter ms_handle_connect 0x7fe5b00b10d0 2024-03-25T19:24:36.795 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.795+0000 7fe5a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:36.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.797+0000 7fe5a5ffb640 10 client.?.objecter ms_handle_connect 0x7fe5b00a9470 2024-03-25T19:24:36.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.797+0000 7fe5a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:36.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.807+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935540356 0 0) 0x7fe5b013e7c0 con 0x7fe5b00b10d0 2024-03-25T19:24:36.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.807+0000 7fe5a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:24:36.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.807+0000 7fe5a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:36.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.807+0000 7fe5a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:36.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.807+0000 7fe5a5ffb640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:24:36.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.807+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594004eb0 con 0x7fe5b00b10d0 2024-03-25T19:24:36.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.817+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088715970 0 0) 0x7fe5b00b0990 con 0x7fe5b00a9470 2024-03-25T19:24:36.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.817+0000 7fe5a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:24:36.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.817+0000 7fe5a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:36.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.817+0000 7fe5a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:36.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.817+0000 7fe5a5ffb640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:24:36.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.817+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5b013e7c0 con 0x7fe5b00a9470 2024-03-25T19:24:36.823 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.824+0000 7fe5a5ffb640 10 client.?.objecter ms_handle_connect 0x7fe5b00b4360 2024-03-25T19:24:36.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.824+0000 7fe5a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.831+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596711636 0 0) 0x7fe594004eb0 con 0x7fe5b00b10d0 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.831+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5b00b0990 con 0x7fe5b00b10d0 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.831+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5a0001690 con 0x7fe5b00b10d0 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.831+0000 7fe5a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.831+0000 7fe5a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.831+0000 7fe5a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:36.831 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:36.832 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:36.832 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:36.832 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:36.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.831+0000 7fe5a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:36.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.838+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257396152 0 0) 0x7fe5b01398c0 con 0x7fe5b00b4360 2024-03-25T19:24:36.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.838+0000 7fe5a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:24:36.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.838+0000 7fe5a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:36.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.838+0000 7fe5a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:36.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.838+0000 7fe5a5ffb640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:24:36.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.838+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe594004eb0 con 0x7fe5b00b4360 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.843+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1243143447 0 0) 0x7fe5b013e7c0 con 0x7fe5b00a9470 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.843+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5b01398c0 con 0x7fe5b00a9470 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.843+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5ac0018a0 con 0x7fe5b00a9470 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.843+0000 7fe5a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.843+0000 7fe5a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.843+0000 7fe5a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:36.843 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:36.844 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:36.844 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:36.853 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3611923043 0 0) 0x7fe5b01398c0 con 0x7fe5b00a9470 2024-03-25T19:24:36.853 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 >> v1:172.21.15.60:6790/0 conn(0x7fe5b00b10d0 legacy=0x7fe5b0138aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 >> v1:172.21.15.60:6789/0 conn(0x7fe5b00b4360 legacy=0x7fe5b01391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 1 monclient: found mon.b 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b013e950 con 0x7fe5b00a9470 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 5 monclient: authenticate success, global_id 4408 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 10 monclient: _renew_subs 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b00ab750 con 0x7fe5b00a9470 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:36.854 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 10 monclient: _renew_subs 2024-03-25T19:24:36.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:36.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b013f5d0 con 0x7fe5b00a9470 2024-03-25T19:24:36.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.853+0000 7fe5b742a640 1 librados: init done 2024-03-25T19:24:36.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.854+0000 7fe5b742a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:36.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.854+0000 7fe5b742a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:36.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.854+0000 7fe5b742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:36.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:36.855+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b013eba0 con 0x7fe5b00a9470 2024-03-25T19:24:37.134 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.135+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5ac0035f0 con 0x7fe5b00a9470 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.135+0000 7fe5a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.135+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5ac0039c0 con 0x7fe5b00a9470 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.135+0000 7fe5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.135+0000 7fe5a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.135+0000 7fe5a5ffb640 10 monclient: dump: 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:37.135 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.135+0000 7fe5a77fe640 4 set_mon_vals no callback set 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.136+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fe5ac003ba0 con 0x7fe5b00a9470 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.136+0000 7fe5b49c5640 10 monclient: get_auth_request con 0x7fe59406df00 auth_method 0 2024-03-25T19:24:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.136+0000 7fe5a5ffb640 10 client.4408.objecter ms_handle_connect 0x7fe59406df00 2024-03-25T19:24:37.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fe5ac053590 con 0x7fe5b00a9470 2024-03-25T19:24:37.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 10 client.4408.objecter ms_dispatch 0x7fe5b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:37.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T19:24:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5ac04f0a0 con 0x7fe5b00a9470 2024-03-25T19:24:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.141+0000 7fe5a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:37.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.400+0000 7fe5b742a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:37.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.400+0000 7fe5b742a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:37.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.400+0000 7fe5b742a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:37.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.400+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5b00627b0 con 0x7fe5b00a9470 2024-03-25T19:24:37.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.430+0000 7fe5a5ffb640 1 -- 172.21.15.60:0/180836254 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fe5ac050fd0 con 0x7fe5b00a9470 2024-03-25T19:24:37.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.430+0000 7fe5a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:37.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.430+0000 7fe5a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:37.431 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:24:37.431 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:37.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.433+0000 7fe5b742a640 10 librados: watch_flush enter 2024-03-25T19:24:37.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.433+0000 7fe5b742a640 10 librados: watch_flush exit 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.433+0000 7fe5b742a640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.433+0000 7fe5b742a640 10 client.4408.objecter successfully canceled tick 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.433+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 >> v1:172.21.15.90:6800/1827638754 conn(0x7fe59406df00 legacy=0x7fe5940703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 10 monclient: shutdown 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 >> v1:172.21.15.90:6789/0 conn(0x7fe5b00a9470 legacy=0x7fe5b01da810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 shutdown_connections 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 >> 172.21.15.60:0/180836254 conn(0x7fe5b00617a0 msgr2=0x7fe5b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 shutdown_connections 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 1 -- 172.21.15.60:0/180836254 wait complete. 2024-03-25T19:24:37.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:37.434+0000 7fe5b742a640 1 librados: shutdown 2024-03-25T19:24:37.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:44.250 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.362+0000 7f30f815d640 10 monclient: get_monmap_and_config 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.362+0000 7f30f815d640 10 monclient: build_initial_monmap 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.362+0000 7f30f815d640 10 monclient: monmap: 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:44.364111+0000 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:44.364111+0000 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:44.363 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:44.364 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:44.364 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:44.364 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.363+0000 7f30f815d640 1 Processor -- start 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.363+0000 7f30f815d640 1 -- start start 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.363+0000 7f30f815d640 10 monclient: init 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.363+0000 7f30f815d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.363+0000 7f30f815d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.363+0000 7f30f815d640 10 monclient(hunting): picked mon.noname-b con 0x7f30f00a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.364+0000 7f30f815d640 10 monclient(hunting): picked mon.noname-c con 0x7f30f013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.364+0000 7f30f815d640 10 monclient(hunting): picked mon.noname-a con 0x7f30f01430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.364+0000 7f30f815d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f0142520 con 0x7f30f01430c0 2024-03-25T19:24:44.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.364+0000 7f30f815d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00b28a0 con 0x7f30f013f710 2024-03-25T19:24:44.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.364+0000 7f30f815d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f00b2a30 con 0x7f30f00a98f0 2024-03-25T19:24:44.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.364+0000 7f30f815d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:44.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.364+0000 7f30f815d640 10 monclient(hunting): authenticate will time out at 6611.462956s 2024-03-25T19:24:44.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.369+0000 7f30f66fa640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f30f01430c0 0x7f30f014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40230/0 (socket says 172.21.15.60:40230) 2024-03-25T19:24:44.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.369+0000 7f30f66fa640 1 -- 172.21.15.60:0/2802757648 learned_addr learned my addr 172.21.15.60:0/2802757648 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:44.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.379+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611751305 0 0) 0x7f30f00b2a30 con 0x7f30f00a98f0 2024-03-25T19:24:44.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.379+0000 7f30f4ef7640 10 monclient(hunting): _negotiate 2024-03-25T19:24:44.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.379+0000 7f30f4ef7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:44.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.379+0000 7f30f4ef7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:44.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.379+0000 7f30f4ef7640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:24:44.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.379+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30dc003520 con 0x7f30f00a98f0 2024-03-25T19:24:44.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.389+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85642437 0 0) 0x7f30f0142520 con 0x7f30f01430c0 2024-03-25T19:24:44.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.389+0000 7f30f4ef7640 10 monclient(hunting): _negotiate 2024-03-25T19:24:44.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.389+0000 7f30f4ef7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:44.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.389+0000 7f30f4ef7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:44.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.389+0000 7f30f4ef7640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:24:44.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.389+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f00b2a30 con 0x7f30f01430c0 2024-03-25T19:24:44.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 557310459 0 0) 0x7f30dc003520 con 0x7f30f00a98f0 2024-03-25T19:24:44.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 >> v1:172.21.15.90:6789/0 conn(0x7f30f013f710 legacy=0x7f30f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:44.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 >> v1:172.21.15.60:6789/0 conn(0x7f30f01430c0 legacy=0x7f30f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:44.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 10 monclient: _finish_hunting 0 2024-03-25T19:24:44.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 1 monclient: found mon.noname-b 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f00b44c0 con 0x7f30f00a98f0 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 10 monclient: _finish_auth 0 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 10 monclient: _check_auth_tickets 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f30ec001690 con 0x7f30f00a98f0 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.395+0000 7f30f4ef7640 10 monclient: dump: 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:44.397 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:44.398 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:44.398 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:44.398 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:44.398 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:44.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.396+0000 7f30f815d640 5 monclient: authenticate success, global_id 4406 2024-03-25T19:24:44.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.396+0000 7f30f815d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:44.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30ec0017f0 con 0x7f30f00a98f0 2024-03-25T19:24:44.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f4ef7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:44.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f4ef7640 1 -- 172.21.15.60:0/2802757648 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f30ec001c00 con 0x7f30f00a98f0 2024-03-25T19:24:44.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f815d640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:44.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f815d640 4 set_mon_vals no callback set 2024-03-25T19:24:44.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f4ef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f4ef7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f4ef7640 10 monclient: dump: 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:44.410 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f815d640 10 monclient: shutdown 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.409+0000 7f30f815d640 1 -- 172.21.15.60:0/2802757648 >> v1:172.21.15.60:6790/0 conn(0x7f30f00a98f0 legacy=0x7f30f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.410+0000 7f30f815d640 1 -- 172.21.15.60:0/2802757648 shutdown_connections 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.410+0000 7f30f815d640 1 -- 172.21.15.60:0/2802757648 >> 172.21.15.60:0/2802757648 conn(0x7f30f00b7ea0 msgr2=0x7f30f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.410+0000 7f30f815d640 1 -- 172.21.15.60:0/2802757648 shutdown_connections 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.410+0000 7f30f815d640 1 -- 172.21.15.60:0/2802757648 wait complete. 2024-03-25T19:24:44.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 10 monclient: build_initial_monmap 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 10 monclient: monmap: 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:44.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 1 librados: starting msgr at 2024-03-25T19:24:44.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 1 librados: starting objecter 2024-03-25T19:24:44.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 1 Processor -- start 2024-03-25T19:24:44.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 1 -- start start 2024-03-25T19:24:44.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 1 librados: setting wanted keys 2024-03-25T19:24:44.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 1 librados: calling monclient init 2024-03-25T19:24:44.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.411+0000 7f30f815d640 10 monclient: init 2024-03-25T19:24:44.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:44.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:24:44.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 10 monclient(hunting): picked mon.noname-c con 0x7f30f00a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:44.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 10 monclient(hunting): picked mon.noname-a con 0x7f30f013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:44.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 10 monclient(hunting): picked mon.noname-b con 0x7f30f01430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:44.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f013be70 con 0x7f30f013f710 2024-03-25T19:24:44.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f0131660 con 0x7f30f01430c0 2024-03-25T19:24:44.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f01317f0 con 0x7f30f00a98f0 2024-03-25T19:24:44.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:44.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.412+0000 7f30f815d640 10 monclient(hunting): authenticate will time out at 6611.511616s 2024-03-25T19:24:44.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.416+0000 7f30f66fa640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f30f01430c0 0x7f30f01e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:44120/0 (socket says 172.21.15.60:44120) 2024-03-25T19:24:44.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.416+0000 7f30f66fa640 1 -- 172.21.15.60:0/1206774948 learned_addr learned my addr 172.21.15.60:0/1206774948 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:44.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.418+0000 7f30e67fc640 10 client.?.objecter ms_handle_connect 0x7f30f01430c0 2024-03-25T19:24:44.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.418+0000 7f30e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:44.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.425+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059610035 0 0) 0x7f30f0131660 con 0x7f30f01430c0 2024-03-25T19:24:44.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.425+0000 7f30e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:44.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.425+0000 7f30e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:44.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.425+0000 7f30e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:44.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.425+0000 7f30e67fc640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:24:44.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.425+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d4004eb0 con 0x7f30f01430c0 2024-03-25T19:24:44.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.431+0000 7f30e67fc640 10 client.?.objecter ms_handle_connect 0x7f30f013f710 2024-03-25T19:24:44.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.431+0000 7f30e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:44.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.434+0000 7f30e67fc640 10 client.?.objecter ms_handle_connect 0x7f30f00a98f0 2024-03-25T19:24:44.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.434+0000 7f30e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.439+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 869781179 0 0) 0x7f30d4004eb0 con 0x7f30f01430c0 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.439+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30f0131660 con 0x7f30f01430c0 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.439+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f30e8001c10 con 0x7f30f01430c0 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.439+0000 7f30e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.439+0000 7f30e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.439+0000 7f30e67fc640 10 monclient(hunting): dump: 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:44.440 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:44.441 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.439+0000 7f30e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:44.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.446+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414489383 0 0) 0x7f30f013be70 con 0x7f30f013f710 2024-03-25T19:24:44.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.446+0000 7f30e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:24:44.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.446+0000 7f30e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:44.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.446+0000 7f30e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:44.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.446+0000 7f30e67fc640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:24:44.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.446+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30d4004eb0 con 0x7f30f013f710 2024-03-25T19:24:44.454 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 649377193 0 0) 0x7f30f0131660 con 0x7f30f01430c0 2024-03-25T19:24:44.454 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 >> v1:172.21.15.60:6790/0 conn(0x7f30f00a98f0 legacy=0x7f30f0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 >> v1:172.21.15.60:6789/0 conn(0x7f30f013f710 legacy=0x7f30f01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 1 monclient: found mon.b 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f0131980 con 0x7f30f01430c0 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 10 monclient: _renew_subs 2024-03-25T19:24:44.455 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f0135c60 con 0x7f30f01430c0 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 10 monclient: _renew_subs 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.453+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f00b7770 con 0x7f30f01430c0 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.454+0000 7f30f815d640 1 librados: init done 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.455+0000 7f30f815d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.455+0000 7f30f815d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.455+0000 7f30f815d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:44.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.455+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f01371d0 con 0x7f30f01430c0 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.747+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30e8004b60 con 0x7f30f01430c0 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.747+0000 7f30e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.747+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f30e8004e70 con 0x7f30f01430c0 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.747+0000 7f30e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.747+0000 7f30e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.747+0000 7f30e67fc640 10 monclient: dump: 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:44.748 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:44.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.747+0000 7f30e7fff640 4 set_mon_vals no callback set 2024-03-25T19:24:44.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.748+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f30e801ae40 con 0x7f30f01430c0 2024-03-25T19:24:44.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.748+0000 7f30f5ef9640 10 monclient: get_auth_request con 0x7f30d406deb0 auth_method 0 2024-03-25T19:24:44.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.749+0000 7f30e67fc640 10 client.4414.objecter ms_handle_connect 0x7f30d406deb0 2024-03-25T19:24:44.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.751+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f30e8054ac0 con 0x7f30f01430c0 2024-03-25T19:24:44.751 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.751+0000 7f30e67fc640 10 client.4414.objecter ms_dispatch 0x7f30f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:44.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.751+0000 7f30e67fc640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:44.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.751+0000 7f30e67fc640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:44.752 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.751+0000 7f30e67fc640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T19:24:44.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.753+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30e80505d0 con 0x7f30f01430c0 2024-03-25T19:24:44.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.753+0000 7f30e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:44.754 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.753+0000 7f30e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:44.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.989+0000 7f30f815d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:44.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.989+0000 7f30f815d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:44.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.989+0000 7f30f815d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:24:44.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:44.989+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30f00627b0 con 0x7f30f01430c0 2024-03-25T19:24:45.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.019+0000 7f30e67fc640 1 -- 172.21.15.60:0/1206774948 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f30e8052500 con 0x7f30f01430c0 2024-03-25T19:24:45.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.019+0000 7f30e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:45.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.019+0000 7f30e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:45.020 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:24:45.020 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 10 librados: watch_flush enter 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 10 librados: watch_flush exit 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 10 client.4414.objecter successfully canceled tick 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 >> v1:172.21.15.90:6800/1827638754 conn(0x7f30d406deb0 legacy=0x7f30d4070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 10 monclient: shutdown 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 >> v1:172.21.15.90:6789/0 conn(0x7f30f01430c0 legacy=0x7f30f01e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 shutdown_connections 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 >> 172.21.15.60:0/1206774948 conn(0x7f30f00b7ea0 msgr2=0x7f30f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 shutdown_connections 2024-03-25T19:24:45.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 1 -- 172.21.15.60:0/1206774948 wait complete. 2024-03-25T19:24:45.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:45.022+0000 7f30f815d640 1 librados: shutdown 2024-03-25T19:24:45.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:51.841 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:51.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.954+0000 7f8136783640 10 monclient: get_monmap_and_config 2024-03-25T19:24:51.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.954+0000 7f8136783640 10 monclient: build_initial_monmap 2024-03-25T19:24:51.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.955+0000 7f8136783640 10 monclient: monmap: 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:51.956680+0000 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:51.956680+0000 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:51.956 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:51.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.955+0000 7f8136783640 1 Processor -- start 2024-03-25T19:24:51.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.955+0000 7f8136783640 1 -- start start 2024-03-25T19:24:51.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.955+0000 7f8136783640 10 monclient: init 2024-03-25T19:24:51.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 10 monclient(hunting): picked mon.noname-a con 0x7f813006c380 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 10 monclient(hunting): picked mon.noname-b con 0x7f813006d670 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 10 monclient(hunting): picked mon.noname-c con 0x7f81300af8f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813006c9a0 con 0x7f813006c380 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81300ada90 con 0x7f81300af8f0 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81300b0d10 con 0x7f813006d670 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:51.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.956+0000 7f8136783640 10 monclient(hunting): authenticate will time out at 6619.055363s 2024-03-25T19:24:51.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.960+0000 7f812ffff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f813006c380 0x7f81300b0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:40660/0 (socket says 172.21.15.60:40660) 2024-03-25T19:24:51.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.960+0000 7f812ffff640 1 -- 172.21.15.60:0/1812569409 learned_addr learned my addr 172.21.15.60:0/1812569409 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:51.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.970+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990890860 0 0) 0x7f81300ada90 con 0x7f81300af8f0 2024-03-25T19:24:51.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.970+0000 7f812effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:51.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.970+0000 7f812effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:51.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.970+0000 7f812effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:51.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.970+0000 7f812effd640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:24:51.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.970+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8118003520 con 0x7f81300af8f0 2024-03-25T19:24:51.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.972+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77992084 0 0) 0x7f813006c9a0 con 0x7f813006c380 2024-03-25T19:24:51.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.972+0000 7f812effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:51.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.972+0000 7f812effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:51.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.972+0000 7f812effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:51.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.972+0000 7f812effd640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:24:51.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.972+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81300ada90 con 0x7f813006c380 2024-03-25T19:24:51.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.976+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094512356 0 0) 0x7f81300b0d10 con 0x7f813006d670 2024-03-25T19:24:51.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.976+0000 7f812effd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:51.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.976+0000 7f812effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:51.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.976+0000 7f812effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:51.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.976+0000 7f812effd640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:24:51.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.976+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f813006c9a0 con 0x7f813006d670 2024-03-25T19:24:51.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4021601731 0 0) 0x7f8118003520 con 0x7f81300af8f0 2024-03-25T19:24:51.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 >> v1:172.21.15.60:6790/0 conn(0x7f813006d670 legacy=0x7f81300ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:51.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 >> v1:172.21.15.60:6789/0 conn(0x7f813006c380 legacy=0x7f81300b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:51.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:51.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 1 monclient: found mon.noname-c 2024-03-25T19:24:51.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:51.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:24:51.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81300add00 con 0x7f81300af8f0 2024-03-25T19:24:51.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 10 monclient: _finish_auth 0 2024-03-25T19:24:51.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:51.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:51.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8128001a10 con 0x7f81300af8f0 2024-03-25T19:24:51.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:51.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:51.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f812effd640 10 monclient: dump: 2024-03-25T19:24:51.987 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:51.988 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:51.988 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:51.988 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:51.988 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:51.988 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:51.988 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:51.988 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:51.989 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:51.989 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:51.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f8136783640 5 monclient: authenticate success, global_id 4417 2024-03-25T19:24:51.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.984+0000 7f8136783640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:51.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8128001b70 con 0x7f81300af8f0 2024-03-25T19:24:51.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f812effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:51.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f8136783640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:51.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f8136783640 4 set_mon_vals no callback set 2024-03-25T19:24:52.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f812effd640 1 -- 172.21.15.60:0/1812569409 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8128001f00 con 0x7f81300af8f0 2024-03-25T19:24:52.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f812effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:52.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f812effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:52.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f812effd640 10 monclient: dump: 2024-03-25T19:24:52.000 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:52.000 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:52.001 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:52.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f8136783640 10 monclient: shutdown 2024-03-25T19:24:52.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f8136783640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:52.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f8136783640 1 -- 172.21.15.60:0/1812569409 >> v1:172.21.15.90:6789/0 conn(0x7f81300af8f0 legacy=0x7f81300aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:52.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f8136783640 1 -- 172.21.15.60:0/1812569409 shutdown_connections 2024-03-25T19:24:52.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.995+0000 7f8136783640 1 -- 172.21.15.60:0/1812569409 >> 172.21.15.60:0/1812569409 conn(0x7f81300b5c00 msgr2=0x7f81300b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:52.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.996+0000 7f8136783640 1 -- 172.21.15.60:0/1812569409 shutdown_connections 2024-03-25T19:24:52.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.996+0000 7f8136783640 1 -- 172.21.15.60:0/1812569409 wait complete. 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.996+0000 7f8136783640 10 monclient: build_initial_monmap 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.996+0000 7f8136783640 10 monclient: monmap: 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:52.003 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:52.004 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:52.004 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:52.004 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:52.004 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:52.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.997+0000 7f8136783640 1 librados: starting msgr at 2024-03-25T19:24:52.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.997+0000 7f8136783640 1 librados: starting objecter 2024-03-25T19:24:52.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.997+0000 7f8136783640 1 Processor -- start 2024-03-25T19:24:52.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.997+0000 7f8136783640 1 -- start start 2024-03-25T19:24:52.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.997+0000 7f8136783640 1 librados: setting wanted keys 2024-03-25T19:24:52.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.997+0000 7f8136783640 1 librados: calling monclient init 2024-03-25T19:24:52.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.997+0000 7f8136783640 10 monclient: init 2024-03-25T19:24:52.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:52.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:52.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 10 monclient(hunting): picked mon.noname-c con 0x7f813006c380 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:52.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 10 monclient(hunting): picked mon.noname-b con 0x7f813006d670 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:52.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 10 monclient(hunting): picked mon.noname-a con 0x7f81300af8f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:52.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8130141e70 con 0x7f81300af8f0 2024-03-25T19:24:52.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81301da9e0 con 0x7f813006d670 2024-03-25T19:24:52.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8130148770 con 0x7f813006c380 2024-03-25T19:24:52.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:52.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:51.998+0000 7f8136783640 10 monclient(hunting): authenticate will time out at 6619.097380s 2024-03-25T19:24:52.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.003+0000 7f812ffff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f813006c380 0x7f8130143ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:38288/0 (socket says 172.21.15.60:38288) 2024-03-25T19:24:52.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.003+0000 7f812ffff640 1 -- 172.21.15.60:0/4244818003 learned_addr learned my addr 172.21.15.60:0/4244818003 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:52.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.004+0000 7f812cff9640 10 client.?.objecter ms_handle_connect 0x7f813006c380 2024-03-25T19:24:52.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.004+0000 7f812cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:52.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.013+0000 7f812cff9640 10 client.?.objecter ms_handle_connect 0x7f813006d670 2024-03-25T19:24:52.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.013+0000 7f812cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:52.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.014+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627536295 0 0) 0x7f8130148770 con 0x7f813006c380 2024-03-25T19:24:52.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.014+0000 7f812cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:52.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.014+0000 7f812cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:52.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.014+0000 7f812cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:52.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.014+0000 7f812cff9640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:24:52.016 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.014+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f811c004eb0 con 0x7f813006c380 2024-03-25T19:24:52.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.016+0000 7f812cff9640 10 client.?.objecter ms_handle_connect 0x7f81300af8f0 2024-03-25T19:24:52.017 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.016+0000 7f812cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:52.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.024+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785183237 0 0) 0x7f81301da9e0 con 0x7f813006d670 2024-03-25T19:24:52.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.024+0000 7f812cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:52.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.024+0000 7f812cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:52.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.024+0000 7f812cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:52.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.024+0000 7f812cff9640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:24:52.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.024+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8130148770 con 0x7f813006d670 2024-03-25T19:24:52.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.036+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437493479 0 0) 0x7f81300add00 con 0x7f81300af8f0 2024-03-25T19:24:52.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.036+0000 7f812cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:24:52.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.036+0000 7f812cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:52.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.036+0000 7f812cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:52.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.036+0000 7f812cff9640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:24:52.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.036+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81301da9e0 con 0x7f81300af8f0 2024-03-25T19:24:52.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.038+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657598157 0 0) 0x7f811c004eb0 con 0x7f813006c380 2024-03-25T19:24:52.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.039+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81300add00 con 0x7f813006c380 2024-03-25T19:24:52.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.039+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f81200019f0 con 0x7f813006c380 2024-03-25T19:24:52.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.039+0000 7f812cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.039+0000 7f812cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.039+0000 7f812cff9640 10 monclient(hunting): dump: 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:52.040 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:52.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.039+0000 7f812cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.055+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 791386836 0 0) 0x7f81301da9e0 con 0x7f81300af8f0 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.055+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f811c004eb0 con 0x7f81300af8f0 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.055+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8128001440 con 0x7f81300af8f0 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.055+0000 7f812cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.055+0000 7f812cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.055+0000 7f812cff9640 10 monclient(hunting): dump: 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.056 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.057 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:52.057 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:52.057 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:52.057 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:52.057 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:52.057 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.056+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929714043 0 0) 0x7f8130148770 con 0x7f813006d670 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.056+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81301da9e0 con 0x7f813006d670 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.056+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8124001870 con 0x7f813006d670 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.056+0000 7f812cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.056+0000 7f812cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.056+0000 7f812cff9640 10 monclient(hunting): dump: 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:52.058 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.059 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.059 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:52.059 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:52.059 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:52.059 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:52.059 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:52.059 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:52.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2912398587 0 0) 0x7f81300add00 con 0x7f813006c380 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 >> v1:172.21.15.90:6789/0 conn(0x7f813006d670 legacy=0x7f81301dae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 >> v1:172.21.15.60:6789/0 conn(0x7f81300af8f0 legacy=0x7f81301d4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 1 monclient: found mon.c 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81301441c0 con 0x7f813006c380 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 10 monclient: _finish_auth 0 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f812cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f8136783640 5 monclient: authenticate success, global_id 4415 2024-03-25T19:24:52.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f8136783640 10 monclient: _renew_subs 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f8136783640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.070+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81301db580 con 0x7f813006c380 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 10 monclient: _renew_subs 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8130143620 con 0x7f813006c380 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 1 librados: init done 2024-03-25T19:24:52.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:52.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:52.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:24:52.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.071+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81300627b0 con 0x7f813006c380 2024-03-25T19:24:52.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.386+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81200047e0 con 0x7f813006c380 2024-03-25T19:24:52.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.386+0000 7f812cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:52.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.386+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8120004950 con 0x7f813006c380 2024-03-25T19:24:52.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.386+0000 7f812cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:52.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.386+0000 7f812cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:52.387 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.386+0000 7f812cff9640 10 monclient: dump: 2024-03-25T19:24:52.387 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:52.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.386+0000 7f812e7fc640 4 set_mon_vals no callback set 2024-03-25T19:24:52.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.387+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f812001a920 con 0x7f813006c380 2024-03-25T19:24:52.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.388+0000 7f812f7fe640 10 monclient: get_auth_request con 0x7f811c06e080 auth_method 0 2024-03-25T19:24:52.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.388+0000 7f812cff9640 10 client.4415.objecter ms_handle_connect 0x7f811c06e080 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f81200546b0 con 0x7f813006c380 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 10 client.4415.objecter ms_dispatch 0x7f8130000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f812004fff0 con 0x7f813006c380 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:52.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.396+0000 7f812cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:24:52.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.650+0000 7f8136783640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:52.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.650+0000 7f8136783640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:52.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.650+0000 7f8136783640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:24:52.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.650+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81301db710 con 0x7f813006c380 2024-03-25T19:24:52.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.787+0000 7f812cff9640 1 -- 172.21.15.60:0/4244818003 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f81301db710 con 0x7f813006c380 2024-03-25T19:24:52.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.787+0000 7f812cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:24:52.788 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.787+0000 7f812cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:24:52.789 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:24:52.789 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:24:52.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 10 librados: watch_flush enter 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 10 librados: watch_flush exit 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 10 client.4415.objecter successfully canceled tick 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 >> v1:172.21.15.90:6800/1827638754 conn(0x7f811c06e080 legacy=0x7f811c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 10 monclient: shutdown 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 >> v1:172.21.15.60:6790/0 conn(0x7f813006c380 legacy=0x7f8130143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 shutdown_connections 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.789+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 >> 172.21.15.60:0/4244818003 conn(0x7f81300b5c00 msgr2=0x7f813013c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.790+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 shutdown_connections 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.790+0000 7f8136783640 1 -- 172.21.15.60:0/4244818003 wait complete. 2024-03-25T19:24:52.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:52.790+0000 7f8136783640 1 librados: shutdown 2024-03-25T19:24:52.802 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:24:54.506 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:04:55.218 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.519 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:04:55.230 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.548 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:04:55.260 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.557 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:55.211 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.567 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:55.221 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.580 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:04:55.291 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.593 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:04:55.304 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.604 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:04:55.258 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:54.619 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:04:55.331 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:24:59.608 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:24:59.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.724+0000 7f2fd877c640 10 monclient: get_monmap_and_config 2024-03-25T19:24:59.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.724+0000 7f2fd877c640 10 monclient: build_initial_monmap 2024-03-25T19:24:59.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.724+0000 7f2fd877c640 10 monclient: monmap: 2024-03-25T19:24:59.724 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:59.724 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:59.724 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:24:59.725735+0000 2024-03-25T19:24:59.725 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:24:59.725735+0000 2024-03-25T19:24:59.725 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:59.725 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:59.725 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:59.725 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:24:59.725 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:24:59.725 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:59.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.724+0000 7f2fd877c640 1 Processor -- start 2024-03-25T19:24:59.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.724+0000 7f2fd877c640 1 -- start start 2024-03-25T19:24:59.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.724+0000 7f2fd877c640 10 monclient: init 2024-03-25T19:24:59.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:59.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:24:59.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd00a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:59.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:59.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd01430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:59.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd0142520 con 0x7f2fd013f710 2024-03-25T19:24:59.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00b28a0 con 0x7f2fd00a98f0 2024-03-25T19:24:59.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00b2a30 con 0x7f2fd01430c0 2024-03-25T19:24:59.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:59.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.725+0000 7f2fd877c640 10 monclient(hunting): authenticate will time out at 6626.824769s 2024-03-25T19:24:59.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.734+0000 7f2fd5d17640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2fd013f710 0x7f2fd01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44550/0 (socket says 172.21.15.60:44550) 2024-03-25T19:24:59.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.734+0000 7f2fd5d17640 1 -- 172.21.15.60:0/2290070686 learned_addr learned my addr 172.21.15.60:0/2290070686 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:59.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.741+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480304844 0 0) 0x7f2fd0142520 con 0x7f2fd013f710 2024-03-25T19:24:59.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.741+0000 7f2fd5516640 10 monclient(hunting): _negotiate 2024-03-25T19:24:59.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.741+0000 7f2fd5516640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:59.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.741+0000 7f2fd5516640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:59.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.741+0000 7f2fd5516640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:24:59.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.741+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fbc003520 con 0x7f2fd013f710 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2618506428 0 0) 0x7f2fbc003520 con 0x7f2fd013f710 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 >> v1:172.21.15.60:6790/0 conn(0x7f2fd01430c0 legacy=0x7f2fd014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 >> v1:172.21.15.90:6789/0 conn(0x7f2fd00a98f0 legacy=0x7f2fd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 10 monclient: _finish_hunting 0 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 1 monclient: found mon.noname-a 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd00b44c0 con 0x7f2fd013f710 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 10 monclient: _finish_auth 0 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 10 monclient: _check_auth_tickets 2024-03-25T19:24:59.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2fc0001670 con 0x7f2fd013f710 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd5516640 10 monclient: dump: 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:59.749 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd877c640 5 monclient: authenticate success, global_id 4434 2024-03-25T19:24:59.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.747+0000 7f2fd877c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc00017d0 con 0x7f2fd013f710 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd5516640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd5516640 1 -- 172.21.15.60:0/2290070686 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2fc0001c20 con 0x7f2fd013f710 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd5516640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd5516640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd5516640 10 monclient: dump: 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:59.753 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:59.754 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:59.754 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:59.754 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:59.754 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd877c640 10 monclient: get_monmap_and_config success 2024-03-25T19:24:59.754 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.752+0000 7f2fd877c640 4 set_mon_vals no callback set 2024-03-25T19:24:59.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.753+0000 7f2fd877c640 10 monclient: shutdown 2024-03-25T19:24:59.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.753+0000 7f2fd877c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:24:59.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.753+0000 7f2fd877c640 1 -- 172.21.15.60:0/2290070686 >> v1:172.21.15.60:6789/0 conn(0x7f2fd013f710 legacy=0x7f2fd01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:59.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.753+0000 7f2fd877c640 1 -- 172.21.15.60:0/2290070686 shutdown_connections 2024-03-25T19:24:59.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.753+0000 7f2fd877c640 1 -- 172.21.15.60:0/2290070686 >> 172.21.15.60:0/2290070686 conn(0x7f2fd00b7ea0 msgr2=0x7f2fd00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:24:59.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.753+0000 7f2fd877c640 1 -- 172.21.15.60:0/2290070686 shutdown_connections 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.753+0000 7f2fd877c640 1 -- 172.21.15.60:0/2290070686 wait complete. 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.754+0000 7f2fd877c640 10 monclient: build_initial_monmap 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.754+0000 7f2fd877c640 10 monclient: monmap: 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:59.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.754+0000 7f2fd877c640 1 librados: starting msgr at 2024-03-25T19:24:59.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.754+0000 7f2fd877c640 1 librados: starting objecter 2024-03-25T19:24:59.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 1 Processor -- start 2024-03-25T19:24:59.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 1 -- start start 2024-03-25T19:24:59.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 1 librados: setting wanted keys 2024-03-25T19:24:59.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 1 librados: calling monclient init 2024-03-25T19:24:59.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient: init 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient: _reopen_session rank -1 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd00a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd01430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00b7d00 con 0x7f2fd01430c0 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd00b7820 con 0x7f2fd013f710 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd0136f10 con 0x7f2fd00a98f0 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient(hunting): _renew_subs 2024-03-25T19:24:59.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.755+0000 7f2fd877c640 10 monclient(hunting): authenticate will time out at 6626.854628s 2024-03-25T19:24:59.770 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.770+0000 7f2fd6d19640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2fd01430c0 0x7f2fd01e2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:44580/0 (socket says 172.21.15.60:44580) 2024-03-25T19:24:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.770+0000 7f2fd6d19640 1 -- 172.21.15.60:0/1422080447 learned_addr learned my addr 172.21.15.60:0/1422080447 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:24:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.771+0000 7f2fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f2fd01430c0 2024-03-25T19:24:59.771 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.771+0000 7f2fc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:59.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.780+0000 7f2fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f2fd00a98f0 2024-03-25T19:24:59.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.780+0000 7f2fc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:59.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.785+0000 7f2fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f2fd013f710 2024-03-25T19:24:59.786 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.785+0000 7f2fc6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:24:59.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.790+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068758051 0 0) 0x7f2fd0136f10 con 0x7f2fd00a98f0 2024-03-25T19:24:59.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.790+0000 7f2fc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:59.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.790+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:59.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.790+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:59.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.790+0000 7f2fc6ffd640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:24:59.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.790+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb4004eb0 con 0x7f2fd00a98f0 2024-03-25T19:24:59.796 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.796+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961715754 0 0) 0x7f2fd00b7820 con 0x7f2fd013f710 2024-03-25T19:24:59.796 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.796+0000 7f2fc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:59.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.796+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:59.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.796+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:59.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.796+0000 7f2fc6ffd640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:24:59.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.796+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd0136f10 con 0x7f2fd013f710 2024-03-25T19:24:59.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.806+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005760845 0 0) 0x7f2fd00b2a30 con 0x7f2fd01430c0 2024-03-25T19:24:59.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.806+0000 7f2fc6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:24:59.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.806+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:24:59.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.806+0000 7f2fc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:24:59.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.806+0000 7f2fc6ffd640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:24:59.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.806+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd00b7820 con 0x7f2fd01430c0 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.810+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598725336 0 0) 0x7f2fd0136f10 con 0x7f2fd013f710 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.810+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fd00b2a30 con 0x7f2fd013f710 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.810+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2fc0001b20 con 0x7f2fd013f710 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.810+0000 7f2fc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.810+0000 7f2fc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.810+0000 7f2fc6ffd640 10 monclient(hunting): dump: 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.811 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:59.812 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:59.812 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:59.812 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:59.812 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:59.812 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:59.812 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.810+0000 7f2fc6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.816+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2502433627 0 0) 0x7f2fb4004eb0 con 0x7f2fd00a98f0 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.816+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fd0136f10 con 0x7f2fd00a98f0 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.816+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2fcc0018c0 con 0x7f2fd00a98f0 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.816+0000 7f2fc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.816+0000 7f2fc6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.816+0000 7f2fc6ffd640 10 monclient(hunting): dump: 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:59.817 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:59.818 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:59.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.819+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 242507176 0 0) 0x7f2fd00b7820 con 0x7f2fd01430c0 2024-03-25T19:24:59.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.819+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb4004eb0 con 0x7f2fd01430c0 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.819+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2fc8001a30 con 0x7f2fd01430c0 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.819+0000 7f2fc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.819+0000 7f2fc6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.819+0000 7f2fc6ffd640 10 monclient(hunting): dump: 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:24:59.820 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2345255776 0 0) 0x7f2fb4004eb0 con 0x7f2fd01430c0 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 >> v1:172.21.15.60:6790/0 conn(0x7f2fd00a98f0 legacy=0x7f2fd0130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 >> v1:172.21.15.90:6789/0 conn(0x7f2fd013f710 legacy=0x7f2fd01dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 1 monclient: found mon.a 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd0135330 con 0x7f2fd01430c0 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:24:59.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 10 monclient: _renew_subs 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd0135780 con 0x7f2fd01430c0 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 10 monclient: _renew_subs 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd013b420 con 0x7f2fd01430c0 2024-03-25T19:24:59.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.836+0000 7f2fd877c640 1 librados: init done 2024-03-25T19:24:59.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.837+0000 7f2fd877c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:59.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.837+0000 7f2fd877c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:24:59.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.837+0000 7f2fd877c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:24:59.839 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:24:59.837+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd0074cc0 con 0x7f2fd01430c0 2024-03-25T19:25:00.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.172+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc8001f20 con 0x7f2fd01430c0 2024-03-25T19:25:00.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.172+0000 7f2fc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:00.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.172+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2fc8003b30 con 0x7f2fd01430c0 2024-03-25T19:25:00.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.172+0000 7f2fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:00.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.172+0000 7f2fc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.172+0000 7f2fc6ffd640 10 monclient: dump: 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.172+0000 7f2fd4d15640 4 set_mon_vals no callback set 2024-03-25T19:25:00.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.173+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2fc8003d50 con 0x7f2fd01430c0 2024-03-25T19:25:00.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.174+0000 7f2fd6518640 10 monclient: get_auth_request con 0x7f2fb406e160 auth_method 0 2024-03-25T19:25:00.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.174+0000 7f2fc6ffd640 10 client.4437.objecter ms_handle_connect 0x7f2fb406e160 2024-03-25T19:25:00.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f2fc8054890 con 0x7f2fd01430c0 2024-03-25T19:25:00.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 10 client.4437.objecter ms_dispatch 0x7f2fd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:00.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:00.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:00.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T19:25:00.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fc80503a0 con 0x7f2fd01430c0 2024-03-25T19:25:00.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:00.181 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.180+0000 7f2fc6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:00.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.429+0000 7f2fd877c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:00.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.429+0000 7f2fd877c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:00.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.429+0000 7f2fd877c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:25:00.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.429+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fd013b770 con 0x7f2fd01430c0 2024-03-25T19:25:00.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.460+0000 7f2fc6ffd640 1 -- 172.21.15.60:0/1422080447 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f2fc80522d0 con 0x7f2fd01430c0 2024-03-25T19:25:00.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.460+0000 7f2fc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:00.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.460+0000 7f2fc6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:00.461 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:00.461 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:00.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 10 librados: watch_flush enter 2024-03-25T19:25:00.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 10 librados: watch_flush exit 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 10 client.4437.objecter successfully canceled tick 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 >> v1:172.21.15.90:6800/1827638754 conn(0x7f2fb406e160 legacy=0x7f2fb4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 10 monclient: shutdown 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 >> v1:172.21.15.60:6789/0 conn(0x7f2fd01430c0 legacy=0x7f2fd01e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 shutdown_connections 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.462+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 >> 172.21.15.60:0/1422080447 conn(0x7f2fd00b7ea0 msgr2=0x7f2fd0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.463+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 shutdown_connections 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.463+0000 7f2fd877c640 1 -- 172.21.15.60:0/1422080447 wait complete. 2024-03-25T19:25:00.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:00.463+0000 7f2fd877c640 1 librados: shutdown 2024-03-25T19:25:00.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:07.281 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:07.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.393+0000 7f5eb8125640 10 monclient: get_monmap_and_config 2024-03-25T19:25:07.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.393+0000 7f5eb8125640 10 monclient: build_initial_monmap 2024-03-25T19:25:07.393 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.393+0000 7f5eb8125640 10 monclient: monmap: 2024-03-25T19:25:07.393 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:07.393 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:07.393 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:25:07.394801+0000 2024-03-25T19:25:07.393 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:25:07.394801+0000 2024-03-25T19:25:07.394 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:07.394 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:07.394 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:07.394 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:25:07.394 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:25:07.394 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:07.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.393+0000 7f5eb8125640 1 Processor -- start 2024-03-25T19:25:07.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.393+0000 7f5eb8125640 1 -- start start 2024-03-25T19:25:07.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.393+0000 7f5eb8125640 10 monclient: init 2024-03-25T19:25:07.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:07.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:07.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 10 monclient(hunting): picked mon.noname-b con 0x7f5eb00a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:07.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:07.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 10 monclient(hunting): picked mon.noname-c con 0x7f5eb01430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:07.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb0142520 con 0x7f5eb013f710 2024-03-25T19:25:07.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb00b28a0 con 0x7f5eb01430c0 2024-03-25T19:25:07.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb00b2a30 con 0x7f5eb00a98f0 2024-03-25T19:25:07.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:07.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.394+0000 7f5eb8125640 10 monclient(hunting): authenticate will time out at 6634.493454s 2024-03-25T19:25:07.403 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.403+0000 7f5eb5ec1640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f5eb00a98f0 0x7f5eb00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:51474/0 (socket says 172.21.15.60:51474) 2024-03-25T19:25:07.403 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.403+0000 7f5eb5ec1640 1 -- 172.21.15.60:0/3134692176 learned_addr learned my addr 172.21.15.60:0/3134692176 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:07.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.417+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537288890 0 0) 0x7f5eb00b2a30 con 0x7f5eb00a98f0 2024-03-25T19:25:07.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.417+0000 7f5eb4ebf640 10 monclient(hunting): _negotiate 2024-03-25T19:25:07.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.417+0000 7f5eb4ebf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:07.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.417+0000 7f5eb4ebf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:07.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.417+0000 7f5eb4ebf640 10 monclient(hunting): my global_id is 4424 2024-03-25T19:25:07.418 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.418+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c003520 con 0x7f5eb00a98f0 2024-03-25T19:25:07.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.421+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742401412 0 0) 0x7f5eb0142520 con 0x7f5eb013f710 2024-03-25T19:25:07.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.421+0000 7f5eb4ebf640 10 monclient(hunting): _negotiate 2024-03-25T19:25:07.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.421+0000 7f5eb4ebf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:07.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.421+0000 7f5eb4ebf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:07.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.421+0000 7f5eb4ebf640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:25:07.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.421+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb00b2a30 con 0x7f5eb013f710 2024-03-25T19:25:07.423 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.423+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784173107 0 0) 0x7f5eb00b28a0 con 0x7f5eb01430c0 2024-03-25T19:25:07.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.423+0000 7f5eb4ebf640 10 monclient(hunting): _negotiate 2024-03-25T19:25:07.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.423+0000 7f5eb4ebf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:07.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.423+0000 7f5eb4ebf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:07.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.423+0000 7f5eb4ebf640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:25:07.424 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.423+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb0142520 con 0x7f5eb01430c0 2024-03-25T19:25:07.432 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2785601023 0 0) 0x7f5eb0142520 con 0x7f5eb01430c0 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 >> v1:172.21.15.60:6790/0 conn(0x7f5eb00a98f0 legacy=0x7f5eb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 >> v1:172.21.15.60:6789/0 conn(0x7f5eb013f710 legacy=0x7f5eb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 10 monclient: _finish_hunting 0 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 1 monclient: found mon.noname-c 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00b44c0 con 0x7f5eb01430c0 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 10 monclient: _finish_auth 0 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 10 monclient: _check_auth_tickets 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5ea8001930 con 0x7f5eb01430c0 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb8125640 5 monclient: authenticate success, global_id 4429 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb8125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:07.433 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb4ebf640 10 monclient: dump: 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:07.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.432+0000 7f5eb8125640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ea8001a90 con 0x7f5eb01430c0 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb4ebf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb4ebf640 1 -- 172.21.15.60:0/3134692176 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5ea8001ee0 con 0x7f5eb01430c0 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb4ebf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb4ebf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb4ebf640 10 monclient: dump: 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:07.443 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:07.444 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:07.444 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:07.444 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:07.444 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb8125640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:07.444 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.442+0000 7f5eb8125640 4 set_mon_vals no callback set 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.443+0000 7f5eb8125640 10 monclient: shutdown 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.443+0000 7f5eb8125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.443+0000 7f5eb8125640 1 -- 172.21.15.60:0/3134692176 >> v1:172.21.15.90:6789/0 conn(0x7f5eb01430c0 legacy=0x7f5eb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.443+0000 7f5eb8125640 1 -- 172.21.15.60:0/3134692176 shutdown_connections 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.443+0000 7f5eb8125640 1 -- 172.21.15.60:0/3134692176 >> 172.21.15.60:0/3134692176 conn(0x7f5eb00b7ea0 msgr2=0x7f5eb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.443+0000 7f5eb8125640 1 -- 172.21.15.60:0/3134692176 shutdown_connections 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.443+0000 7f5eb8125640 1 -- 172.21.15.60:0/3134692176 wait complete. 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.444+0000 7f5eb8125640 10 monclient: build_initial_monmap 2024-03-25T19:25:07.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 10 monclient: monmap: 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:25:07.446 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 1 librados: starting msgr at 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 1 librados: starting objecter 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 1 Processor -- start 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 1 -- start start 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 1 librados: setting wanted keys 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 1 librados: calling monclient init 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.445+0000 7f5eb8125640 10 monclient: init 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:07.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 10 monclient(hunting): picked mon.noname-a con 0x7f5eb00a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 10 monclient(hunting): picked mon.noname-c con 0x7f5eb013f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 10 monclient(hunting): picked mon.noname-b con 0x7f5eb01430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb00abe50 con 0x7f5eb00a98f0 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb0138f20 con 0x7f5eb01430c0 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.446+0000 7f5eb8125640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb01d51f0 con 0x7f5eb013f710 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.447+0000 7f5eb8125640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:07.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.447+0000 7f5eb8125640 10 monclient(hunting): authenticate will time out at 6634.545925s 2024-03-25T19:25:07.456 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.456+0000 7f5eb66c2640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f5eb01430c0 0x7f5eb01e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:58136/0 (socket says 172.21.15.60:58136) 2024-03-25T19:25:07.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.456+0000 7f5eb66c2640 1 -- 172.21.15.60:0/1618242286 learned_addr learned my addr 172.21.15.60:0/1618242286 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:07.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.456+0000 7f5ea67fc640 10 client.?.objecter ms_handle_connect 0x7f5eb01430c0 2024-03-25T19:25:07.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.457+0000 7f5ea67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:07.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.459+0000 7f5ea67fc640 10 client.?.objecter ms_handle_connect 0x7f5eb013f710 2024-03-25T19:25:07.459 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.459+0000 7f5ea67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:07.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.468+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516587749 0 0) 0x7f5eb0138f20 con 0x7f5eb01430c0 2024-03-25T19:25:07.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.468+0000 7f5ea67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:07.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.468+0000 7f5ea67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:07.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.468+0000 7f5ea67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:07.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.468+0000 7f5ea67fc640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:25:07.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.468+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e94004eb0 con 0x7f5eb01430c0 2024-03-25T19:25:07.476 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.475+0000 7f5ea67fc640 10 client.?.objecter ms_handle_connect 0x7f5eb00a98f0 2024-03-25T19:25:07.476 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.475+0000 7f5ea67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:07.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.479+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341886607 0 0) 0x7f5eb01d51f0 con 0x7f5eb013f710 2024-03-25T19:25:07.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.479+0000 7f5ea67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:07.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.479+0000 7f5ea67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:07.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.479+0000 7f5ea67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:07.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.479+0000 7f5ea67fc640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:25:07.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.479+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb0138f20 con 0x7f5eb013f710 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.482+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3780569628 0 0) 0x7f5e94004eb0 con 0x7f5eb01430c0 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.482+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb01d51f0 con 0x7f5eb01430c0 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.482+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5ea8001ea0 con 0x7f5eb01430c0 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.482+0000 7f5ea67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.482+0000 7f5ea67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.482+0000 7f5ea67fc640 10 monclient(hunting): dump: 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.483 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:07.484 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:07.484 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:07.484 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:07.484 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:07.484 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:07.484 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.482+0000 7f5ea67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:07.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.496+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147976948 0 0) 0x7f5eb00abe50 con 0x7f5eb00a98f0 2024-03-25T19:25:07.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.496+0000 7f5ea67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:07.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.496+0000 7f5ea67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:07.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.496+0000 7f5ea67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:07.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.496+0000 7f5ea67fc640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:25:07.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.496+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e94004eb0 con 0x7f5eb00a98f0 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.506+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 380011302 0 0) 0x7f5eb0138f20 con 0x7f5eb013f710 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.506+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5eb00abe50 con 0x7f5eb013f710 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.506+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5ea0001930 con 0x7f5eb013f710 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.506+0000 7f5ea67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.506+0000 7f5ea67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.506+0000 7f5ea67fc640 10 monclient(hunting): dump: 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:07.507 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:07.508 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:07.508 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:07.508 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:07.508 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1766600284 0 0) 0x7f5eb01d51f0 con 0x7f5eb01430c0 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 >> v1:172.21.15.60:6790/0 conn(0x7f5eb013f710 legacy=0x7f5eb01d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 >> v1:172.21.15.60:6789/0 conn(0x7f5eb00a98f0 legacy=0x7f5eb0138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 1 monclient: found mon.b 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb01d7730 con 0x7f5eb01430c0 2024-03-25T19:25:07.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 10 monclient: _finish_auth 0 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5ea67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 10 monclient: _renew_subs 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb01d8360 con 0x7f5eb01430c0 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 10 monclient: _renew_subs 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb0137ce0 con 0x7f5eb01430c0 2024-03-25T19:25:07.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.512+0000 7f5eb8125640 1 librados: init done 2024-03-25T19:25:07.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.513+0000 7f5eb8125640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:07.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.513+0000 7f5eb8125640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:07.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.513+0000 7f5eb8125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:07.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.513+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb01d7590 con 0x7f5eb01430c0 2024-03-25T19:25:07.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.816+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ea8003c20 con 0x7f5eb01430c0 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.816+0000 7f5ea67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.816+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5ea8002100 con 0x7f5eb01430c0 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.816+0000 7f5ea67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.816+0000 7f5ea67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.816+0000 7f5ea67fc640 10 monclient: dump: 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:07.817 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:07.818 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:07.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.816+0000 7f5ea7fff640 4 set_mon_vals no callback set 2024-03-25T19:25:07.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.817+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f5ea8003ed0 con 0x7f5eb01430c0 2024-03-25T19:25:07.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.817+0000 7f5eb5ec1640 10 monclient: get_auth_request con 0x7f5e9406dfb0 auth_method 0 2024-03-25T19:25:07.818 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.817+0000 7f5ea67fc640 10 client.4432.objecter ms_handle_connect 0x7f5e9406dfb0 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f5ea8054a40 con 0x7f5eb01430c0 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 10 client.4432.objecter ms_dispatch 0x7f5eb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ea8050550 con 0x7f5eb01430c0 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:07.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:07.823+0000 7f5ea67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:08.045 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.044+0000 7f5eb8125640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:08.045 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.044+0000 7f5eb8125640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:08.045 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.044+0000 7f5eb8125640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:08.045 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.044+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eb00627b0 con 0x7f5eb01430c0 2024-03-25T19:25:08.079 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.079+0000 7f5ea67fc640 1 -- 172.21.15.60:0/1618242286 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f5ea8052480 con 0x7f5eb01430c0 2024-03-25T19:25:08.080 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.079+0000 7f5ea67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:08.080 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.079+0000 7f5ea67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:08.080 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:08.080 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:08.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 10 librados: watch_flush enter 2024-03-25T19:25:08.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 10 librados: watch_flush exit 2024-03-25T19:25:08.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T19:25:08.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 10 client.4432.objecter successfully canceled tick 2024-03-25T19:25:08.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 >> v1:172.21.15.90:6800/1827638754 conn(0x7f5e9406dfb0 legacy=0x7f5e94070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 10 monclient: shutdown 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 >> v1:172.21.15.90:6789/0 conn(0x7f5eb01430c0 legacy=0x7f5eb01e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 shutdown_connections 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 >> 172.21.15.60:0/1618242286 conn(0x7f5eb00b7ea0 msgr2=0x7f5eb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 shutdown_connections 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 1 -- 172.21.15.60:0/1618242286 wait complete. 2024-03-25T19:25:08.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:08.081+0000 7f5eb8125640 1 librados: shutdown 2024-03-25T19:25:08.093 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:14.570 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:05:15.281 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.573 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:05:15.284 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.616 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:05:15.328 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.629 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:05:15.341 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.639 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:15.293 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.641 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:05:15.352 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.643 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:15.297 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.665 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:15.319 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.671 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:05:15.383 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:14.898 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:15.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.011+0000 7f3622888640 10 monclient: get_monmap_and_config 2024-03-25T19:25:15.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.011+0000 7f3622888640 10 monclient: build_initial_monmap 2024-03-25T19:25:15.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.011+0000 7f3622888640 10 monclient: monmap: 2024-03-25T19:25:15.010 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:15.010 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:15.010 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:25:15.011801+0000 2024-03-25T19:25:15.011 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:25:15.011801+0000 2024-03-25T19:25:15.011 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:15.011 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:15.011 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:15.011 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:25:15.011 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:25:15.011 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:15.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.011+0000 7f3622888640 1 Processor -- start 2024-03-25T19:25:15.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.011+0000 7f3622888640 1 -- start start 2024-03-25T19:25:15.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.011+0000 7f3622888640 10 monclient: init 2024-03-25T19:25:15.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:15.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:15.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 10 monclient(hunting): picked mon.noname-c con 0x7f361c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:15.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 10 monclient(hunting): picked mon.noname-b con 0x7f361c0b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:15.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 10 monclient(hunting): picked mon.noname-a con 0x7f361c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:15.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c0af940 con 0x7f361c14be90 2024-03-25T19:25:15.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c0b0a70 con 0x7f361c0b4d60 2024-03-25T19:25:15.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c0b0d50 con 0x7f361c0b1730 2024-03-25T19:25:15.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:15.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.012+0000 7f3622888640 10 monclient(hunting): authenticate will time out at 6642.110389s 2024-03-25T19:25:15.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.014+0000 7f361bfff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f361c0b4d60 0x7f361c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:34508/0 (socket says 172.21.15.60:34508) 2024-03-25T19:25:15.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.015+0000 7f361bfff640 1 -- 172.21.15.60:0/491256738 learned_addr learned my addr 172.21.15.60:0/491256738 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:15.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.025+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858785389 0 0) 0x7f361c0b0a70 con 0x7f361c0b4d60 2024-03-25T19:25:15.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.025+0000 7f361affd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:15.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.025+0000 7f361affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:15.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.025+0000 7f361affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:15.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.025+0000 7f361affd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:25:15.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.025+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3604003520 con 0x7f361c0b4d60 2024-03-25T19:25:15.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.028+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60743730 0 0) 0x7f361c0af940 con 0x7f361c14be90 2024-03-25T19:25:15.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.028+0000 7f361affd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:15.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.028+0000 7f361affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:15.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.028+0000 7f361affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:15.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.028+0000 7f361affd640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:25:15.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.028+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c0b0a70 con 0x7f361c14be90 2024-03-25T19:25:15.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 740172921 0 0) 0x7f3604003520 con 0x7f361c0b4d60 2024-03-25T19:25:15.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 >> v1:172.21.15.60:6790/0 conn(0x7f361c0b1730 legacy=0x7f361c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:15.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 >> v1:172.21.15.60:6789/0 conn(0x7f361c14be90 legacy=0x7f361c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:15.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 1 monclient: found mon.noname-c 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c0b1030 con 0x7f361c0b4d60 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 10 monclient: _finish_auth 0 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f360c001690 con 0x7f361c0b4d60 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f3622888640 5 monclient: authenticate success, global_id 4435 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f3622888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f361affd640 10 monclient: dump: 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:15.041 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:15.042 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.041+0000 7f3622888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:15.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f360c0017f0 con 0x7f361c0b4d60 2024-03-25T19:25:15.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f361affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f361affd640 1 -- 172.21.15.60:0/491256738 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f360c001c00 con 0x7f361c0b4d60 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f3622888640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f3622888640 4 set_mon_vals no callback set 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f361affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f361affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f361affd640 10 monclient: dump: 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:15.069 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:15.070 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:15.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f3622888640 10 monclient: shutdown 2024-03-25T19:25:15.071 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f3622888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.069+0000 7f3622888640 1 -- 172.21.15.60:0/491256738 >> v1:172.21.15.90:6789/0 conn(0x7f361c0b4d60 legacy=0x7f361c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.070+0000 7f3622888640 1 -- 172.21.15.60:0/491256738 shutdown_connections 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.070+0000 7f3622888640 1 -- 172.21.15.60:0/491256738 >> 172.21.15.60:0/491256738 conn(0x7f361c0617a0 msgr2=0x7f361c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.070+0000 7f3622888640 1 -- 172.21.15.60:0/491256738 shutdown_connections 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.070+0000 7f3622888640 1 -- 172.21.15.60:0/491256738 wait complete. 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.070+0000 7f3622888640 10 monclient: build_initial_monmap 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.070+0000 7f3622888640 10 monclient: monmap: 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:15.072 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.071+0000 7f3622888640 1 librados: starting msgr at 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.071+0000 7f3622888640 1 librados: starting objecter 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.071+0000 7f3622888640 1 Processor -- start 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.071+0000 7f3622888640 1 -- start start 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.071+0000 7f3622888640 1 librados: setting wanted keys 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.071+0000 7f3622888640 1 librados: calling monclient init 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.071+0000 7f3622888640 10 monclient: init 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 10 monclient(hunting): picked mon.noname-b con 0x7f361c0b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 10 monclient(hunting): picked mon.noname-c con 0x7f361c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:15.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 10 monclient(hunting): picked mon.noname-a con 0x7f361c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:15.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c13bf50 con 0x7f361c14be90 2024-03-25T19:25:15.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c1db000 con 0x7f361c0b1730 2024-03-25T19:25:15.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361c1399a0 con 0x7f361c0b4d60 2024-03-25T19:25:15.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:15.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.072+0000 7f3622888640 10 monclient(hunting): authenticate will time out at 6642.170149s 2024-03-25T19:25:15.080 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.081+0000 7f361bfff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f361c0b1730 0x7f361c142de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:34542/0 (socket says 172.21.15.60:34542) 2024-03-25T19:25:15.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.081+0000 7f361bfff640 1 -- 172.21.15.60:0/2140017125 learned_addr learned my addr 172.21.15.60:0/2140017125 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:15.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.084+0000 7f3618ff9640 10 client.?.objecter ms_handle_connect 0x7f361c0b1730 2024-03-25T19:25:15.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.084+0000 7f3618ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:15.084 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.085+0000 7f3618ff9640 10 client.?.objecter ms_handle_connect 0x7f361c0b4d60 2024-03-25T19:25:15.084 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.085+0000 7f3618ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:15.084 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.085+0000 7f3618ff9640 10 client.?.objecter ms_handle_connect 0x7f361c14be90 2024-03-25T19:25:15.084 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.085+0000 7f3618ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:15.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.091+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752348398 0 0) 0x7f361c1db000 con 0x7f361c0b1730 2024-03-25T19:25:15.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.092+0000 7f3618ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:15.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.092+0000 7f3618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:15.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.092+0000 7f3618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:15.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.092+0000 7f3618ff9640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:25:15.091 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.092+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3608004eb0 con 0x7f361c0b1730 2024-03-25T19:25:15.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.097+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619020238 0 0) 0x7f361c1399a0 con 0x7f361c0b4d60 2024-03-25T19:25:15.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.097+0000 7f3618ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:15.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.097+0000 7f3618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:15.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.097+0000 7f3618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:15.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.097+0000 7f3618ff9640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:25:15.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.097+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c1db000 con 0x7f361c0b4d60 2024-03-25T19:25:15.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.099+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265112423 0 0) 0x7f361c13bf50 con 0x7f361c14be90 2024-03-25T19:25:15.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.099+0000 7f3618ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:15.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.099+0000 7f3618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:15.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.099+0000 7f3618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:15.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.099+0000 7f3618ff9640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:25:15.099 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.099+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c1399a0 con 0x7f361c14be90 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.113+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961213747 0 0) 0x7f3608004eb0 con 0x7f361c0b1730 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.113+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f361c13bf50 con 0x7f361c0b1730 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.113+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f360c001bc0 con 0x7f361c0b1730 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.113+0000 7f3618ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.113+0000 7f3618ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.113+0000 7f3618ff9640 10 monclient(hunting): dump: 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:15.113 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:15.114 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:15.114 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:15.114 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:15.114 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:15.114 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.113+0000 7f3618ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:15.126 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 871351912 0 0) 0x7f361c13bf50 con 0x7f361c0b1730 2024-03-25T19:25:15.126 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 >> v1:172.21.15.60:6790/0 conn(0x7f361c0b4d60 legacy=0x7f361c139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:15.126 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 >> v1:172.21.15.60:6789/0 conn(0x7f361c14be90 legacy=0x7f361c1db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:15.126 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 1 monclient: found mon.b 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361c139b30 con 0x7f361c0b1730 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 10 monclient: _finish_auth 0 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3618ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3622888640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3622888640 10 monclient: _renew_subs 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3622888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.126+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361c1dbc30 con 0x7f361c0b1730 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.127+0000 7f3622888640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.127+0000 7f3622888640 10 monclient: _renew_subs 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.127+0000 7f3622888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:15.127 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.127+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f361c1e3400 con 0x7f361c0b1730 2024-03-25T19:25:15.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.127+0000 7f3622888640 1 librados: init done 2024-03-25T19:25:15.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.128+0000 7f3622888640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:15.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.128+0000 7f3622888640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:15.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.128+0000 7f3622888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:15.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.128+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361c1db1b0 con 0x7f361c0b1730 2024-03-25T19:25:15.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.419+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f360c0011f0 con 0x7f361c0b1730 2024-03-25T19:25:15.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.419+0000 7f3618ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:15.419 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.419+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f360c004a90 con 0x7f361c0b1730 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.419+0000 7f3618ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.419+0000 7f3618ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.419+0000 7f3618ff9640 10 monclient: dump: 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:15.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.419+0000 7f361a7fc640 4 set_mon_vals no callback set 2024-03-25T19:25:15.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.420+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f360c01aae0 con 0x7f361c0b1730 2024-03-25T19:25:15.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.421+0000 7f361b7fe640 10 monclient: get_auth_request con 0x7f360806dda0 auth_method 0 2024-03-25T19:25:15.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.421+0000 7f3618ff9640 10 client.4438.objecter ms_handle_connect 0x7f360806dda0 2024-03-25T19:25:15.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.426+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f360c0547d0 con 0x7f361c0b1730 2024-03-25T19:25:15.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.426+0000 7f3618ff9640 10 client.4438.objecter ms_dispatch 0x7f361c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:15.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.426+0000 7f3618ff9640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:15.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.426+0000 7f3618ff9640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:15.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.426+0000 7f3618ff9640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T19:25:15.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.427+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f360c050110 con 0x7f361c0b1730 2024-03-25T19:25:15.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.427+0000 7f3618ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:15.427 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.427+0000 7f3618ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:15.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.675+0000 7f3622888640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:15.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.675+0000 7f3622888640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:15.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.675+0000 7f3622888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:15.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.675+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f361c0627b0 con 0x7f361c0b1730 2024-03-25T19:25:15.703 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.704+0000 7f3618ff9640 1 -- 172.21.15.60:0/2140017125 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f360c052210 con 0x7f361c0b1730 2024-03-25T19:25:15.703 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.704+0000 7f3618ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:15.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.704+0000 7f3618ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:15.704 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:15.704 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 10 librados: watch_flush enter 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 10 librados: watch_flush exit 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 10 client.4438.objecter successfully canceled tick 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 >> v1:172.21.15.90:6800/1827638754 conn(0x7f360806dda0 legacy=0x7f3608070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 10 monclient: shutdown 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 >> v1:172.21.15.90:6789/0 conn(0x7f361c0b1730 legacy=0x7f361c142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:15.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 shutdown_connections 2024-03-25T19:25:15.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 >> 172.21.15.60:0/2140017125 conn(0x7f361c0617a0 msgr2=0x7f361c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:15.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 shutdown_connections 2024-03-25T19:25:15.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 1 -- 172.21.15.60:0/2140017125 wait complete. 2024-03-25T19:25:15.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:15.706+0000 7f3622888640 1 librados: shutdown 2024-03-25T19:25:15.717 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:22.523 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:22.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.624+0000 7fa9ae0f4640 10 monclient: get_monmap_and_config 2024-03-25T19:25:22.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.624+0000 7fa9ae0f4640 10 monclient: build_initial_monmap 2024-03-25T19:25:22.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.624+0000 7fa9ae0f4640 10 monclient: monmap: 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:25:22.625026+0000 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:25:22.625026+0000 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:22.624 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.625+0000 7fa9ae0f4640 1 Processor -- start 2024-03-25T19:25:22.625 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.625+0000 7fa9ae0f4640 1 -- start start 2024-03-25T19:25:22.625 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.625+0000 7fa9ae0f4640 10 monclient: init 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 10 monclient(hunting): picked mon.noname-a con 0x7fa9a80b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 10 monclient(hunting): picked mon.noname-c con 0x7fa9a80b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 10 monclient(hunting): picked mon.noname-b con 0x7fa9a814be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a80af940 con 0x7fa9a80b4d60 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a80b0a70 con 0x7fa9a80b1730 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a80b0d50 con 0x7fa9a814be90 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:22.626 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.626+0000 7fa9ae0f4640 10 monclient(hunting): authenticate will time out at 6649.724034s 2024-03-25T19:25:22.639 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.640+0000 7fa9a77fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa9a80b4d60 0x7fa9a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:41724/0 (socket says 172.21.15.60:41724) 2024-03-25T19:25:22.639 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.640+0000 7fa9a77fe640 1 -- 172.21.15.60:0/763106380 learned_addr learned my addr 172.21.15.60:0/763106380 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:22.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.652+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015272692 0 0) 0x7fa9a80b0a70 con 0x7fa9a80b1730 2024-03-25T19:25:22.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.652+0000 7fa9a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:22.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.652+0000 7fa9a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:22.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.652+0000 7fa9a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:22.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.652+0000 7fa9a67fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:25:22.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.652+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa990003520 con 0x7fa9a80b1730 2024-03-25T19:25:22.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.654+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118137001 0 0) 0x7fa9a80af940 con 0x7fa9a80b4d60 2024-03-25T19:25:22.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.654+0000 7fa9a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:22.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.654+0000 7fa9a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:22.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.654+0000 7fa9a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:22.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.654+0000 7fa9a67fc640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:25:22.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.654+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a80b0a70 con 0x7fa9a80b4d60 2024-03-25T19:25:22.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.665+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978734275 0 0) 0x7fa9a80b0d50 con 0x7fa9a814be90 2024-03-25T19:25:22.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:25:22.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:22.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:22.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:25:22.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a80af940 con 0x7fa9a814be90 2024-03-25T19:25:22.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2302644591 0 0) 0x7fa9a80b0a70 con 0x7fa9a80b4d60 2024-03-25T19:25:22.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 >> v1:172.21.15.60:6790/0 conn(0x7fa9a814be90 legacy=0x7fa9a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 >> v1:172.21.15.90:6789/0 conn(0x7fa9a80b1730 legacy=0x7fa9a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 1 monclient: found mon.noname-a 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a80b1030 con 0x7fa9a80b4d60 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient: _finish_auth 0 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa998001670 con 0x7fa9a80b4d60 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9ae0f4640 5 monclient: authenticate success, global_id 4452 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9ae0f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9a67fc640 10 monclient: dump: 2024-03-25T19:25:22.667 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:22.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.666+0000 7fa9ae0f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9980017d0 con 0x7fa9a80b4d60 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9a67fc640 1 -- 172.21.15.60:0/763106380 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa998001be0 con 0x7fa9a80b4d60 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9ae0f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9ae0f4640 4 set_mon_vals no callback set 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.675+0000 7fa9a67fc640 10 monclient: dump: 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:22.675 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:22.676 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:22.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.676+0000 7fa9ae0f4640 10 monclient: shutdown 2024-03-25T19:25:22.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.676+0000 7fa9ae0f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:22.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.676+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/763106380 >> v1:172.21.15.60:6789/0 conn(0x7fa9a80b4d60 legacy=0x7fa9a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.676+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/763106380 shutdown_connections 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.676+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/763106380 >> 172.21.15.60:0/763106380 conn(0x7fa9a80617a0 msgr2=0x7fa9a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.676+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/763106380 shutdown_connections 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.676+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/763106380 wait complete. 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.677+0000 7fa9ae0f4640 10 monclient: build_initial_monmap 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.677+0000 7fa9ae0f4640 10 monclient: monmap: 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:25:22.678 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:22.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.677+0000 7fa9ae0f4640 1 librados: starting msgr at 2024-03-25T19:25:22.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.677+0000 7fa9ae0f4640 1 librados: starting objecter 2024-03-25T19:25:22.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.677+0000 7fa9ae0f4640 1 Processor -- start 2024-03-25T19:25:22.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.678+0000 7fa9ae0f4640 1 -- start start 2024-03-25T19:25:22.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.678+0000 7fa9ae0f4640 1 librados: setting wanted keys 2024-03-25T19:25:22.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.678+0000 7fa9ae0f4640 1 librados: calling monclient init 2024-03-25T19:25:22.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.678+0000 7fa9ae0f4640 10 monclient: init 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.678+0000 7fa9ae0f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.678+0000 7fa9ae0f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 10 monclient(hunting): picked mon.noname-c con 0x7fa9a80b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 10 monclient(hunting): picked mon.noname-a con 0x7fa9a80b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 10 monclient(hunting): picked mon.noname-b con 0x7fa9a814be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a8136720 con 0x7fa9a80b4d60 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a81368b0 con 0x7fa9a814be90 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a81383d0 con 0x7fa9a80b1730 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:22.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.679+0000 7fa9ae0f4640 10 monclient(hunting): authenticate will time out at 6649.777149s 2024-03-25T19:25:22.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.691+0000 7fa9a7fff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa9a814be90 0x7fa9a81e2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:43742/0 (socket says 172.21.15.60:43742) 2024-03-25T19:25:22.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.691+0000 7fa9a7fff640 1 -- 172.21.15.60:0/1182213271 learned_addr learned my addr 172.21.15.60:0/1182213271 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:22.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.692+0000 7fa98ffff640 10 client.?.objecter ms_handle_connect 0x7fa9a814be90 2024-03-25T19:25:22.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.692+0000 7fa98ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:22.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.700+0000 7fa98ffff640 10 client.?.objecter ms_handle_connect 0x7fa9a80b4d60 2024-03-25T19:25:22.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.700+0000 7fa98ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 10 client.?.objecter ms_handle_connect 0x7fa9a80b1730 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451196193 0 0) 0x7fa9a81368b0 con 0x7fa9a814be90 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:25:22.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.701+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa994004eb0 con 0x7fa9a814be90 2024-03-25T19:25:22.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.717+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927088727 0 0) 0x7fa9a8136720 con 0x7fa9a80b4d60 2024-03-25T19:25:22.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.717+0000 7fa98ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:22.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.717+0000 7fa98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:22.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.717+0000 7fa98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:22.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.717+0000 7fa98ffff640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:25:22.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.717+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a81368b0 con 0x7fa9a80b4d60 2024-03-25T19:25:22.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.719+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914667970 0 0) 0x7fa9a81383d0 con 0x7fa9a80b1730 2024-03-25T19:25:22.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.719+0000 7fa98ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:22.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.719+0000 7fa98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:22.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.719+0000 7fa98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:22.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.719+0000 7fa98ffff640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:25:22.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.719+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a8136720 con 0x7fa9a80b1730 2024-03-25T19:25:22.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.729+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116787398 0 0) 0x7fa994004eb0 con 0x7fa9a814be90 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.729+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9a81383d0 con 0x7fa9a814be90 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.729+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa9a0001c70 con 0x7fa9a814be90 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.729+0000 7fa98ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.729+0000 7fa98ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.729+0000 7fa98ffff640 10 monclient(hunting): dump: 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:22.729 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:22.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.729+0000 7fa98ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2975323468 0 0) 0x7fa9a81383d0 con 0x7fa9a814be90 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 >> v1:172.21.15.60:6790/0 conn(0x7fa9a80b1730 legacy=0x7fa9a812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 >> v1:172.21.15.60:6789/0 conn(0x7fa9a80b4d60 legacy=0x7fa9a81d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 1 monclient: found mon.b 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8138560 con 0x7fa9a814be90 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 10 monclient: _finish_auth 0 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa98ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:22.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 10 monclient: _renew_subs 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a8136bb0 con 0x7fa9a814be90 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 10 monclient: _renew_subs 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a813ee20 con 0x7fa9a814be90 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.743+0000 7fa9ae0f4640 1 librados: init done 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.744+0000 7fa9ae0f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:22.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.744+0000 7fa9ae0f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:22.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.744+0000 7fa9ae0f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:22.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:22.744+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a8136eb0 con 0x7fa9a814be90 2024-03-25T19:25:23.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.023+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9a0003970 con 0x7fa9a814be90 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.023+0000 7fa98ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.023+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa9a0003d80 con 0x7fa9a814be90 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.023+0000 7fa98ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.023+0000 7fa98ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.023+0000 7fa98ffff640 10 monclient: dump: 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:23.023 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:23.024 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:23.024 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:23.024 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:23.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.023+0000 7fa9a5ffb640 4 set_mon_vals no callback set 2024-03-25T19:25:23.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.024+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fa9a0003fa0 con 0x7fa9a814be90 2024-03-25T19:25:23.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.024+0000 7fa9a77fe640 10 monclient: get_auth_request con 0x7fa99406ddf0 auth_method 0 2024-03-25T19:25:23.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.025+0000 7fa98ffff640 10 client.4444.objecter ms_handle_connect 0x7fa99406ddf0 2024-03-25T19:25:23.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.029+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fa9a0054a60 con 0x7fa9a814be90 2024-03-25T19:25:23.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.029+0000 7fa98ffff640 10 client.4444.objecter ms_dispatch 0x7fa9a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:23.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.029+0000 7fa98ffff640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:23.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.029+0000 7fa98ffff640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:23.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.029+0000 7fa98ffff640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T19:25:23.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.030+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9a0050570 con 0x7fa9a814be90 2024-03-25T19:25:23.029 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.030+0000 7fa98ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:23.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.030+0000 7fa98ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:23.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.276+0000 7fa9ae0f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:23.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.276+0000 7fa9ae0f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:23.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.276+0000 7fa9ae0f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:23.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.276+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9a80627b0 con 0x7fa9a814be90 2024-03-25T19:25:23.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.311+0000 7fa98ffff640 1 -- 172.21.15.60:0/1182213271 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fa9a00524a0 con 0x7fa9a814be90 2024-03-25T19:25:23.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.311+0000 7fa98ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:23.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.311+0000 7fa98ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:23.311 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:23.311 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:23.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.313+0000 7fa9ae0f4640 10 librados: watch_flush enter 2024-03-25T19:25:23.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 10 librados: watch_flush exit 2024-03-25T19:25:23.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T19:25:23.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 10 client.4444.objecter successfully canceled tick 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 >> v1:172.21.15.90:6800/1827638754 conn(0x7fa99406ddf0 legacy=0x7fa9940702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 10 monclient: shutdown 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 >> v1:172.21.15.90:6789/0 conn(0x7fa9a814be90 legacy=0x7fa9a81e2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 shutdown_connections 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 >> 172.21.15.60:0/1182213271 conn(0x7fa9a80617a0 msgr2=0x7fa9a813c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 shutdown_connections 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 1 -- 172.21.15.60:0/1182213271 wait complete. 2024-03-25T19:25:23.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:23.314+0000 7fa9ae0f4640 1 librados: shutdown 2024-03-25T19:25:23.327 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:30.133 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:30.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.250+0000 7fe5f2dbc640 10 monclient: get_monmap_and_config 2024-03-25T19:25:30.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.250+0000 7fe5f2dbc640 10 monclient: build_initial_monmap 2024-03-25T19:25:30.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.250+0000 7fe5f2dbc640 10 monclient: monmap: 2024-03-25T19:25:30.250 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:25:30.251592+0000 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:25:30.251592+0000 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:25:30.251 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:30.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.251+0000 7fe5f2dbc640 1 Processor -- start 2024-03-25T19:25:30.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.251+0000 7fe5f2dbc640 1 -- start start 2024-03-25T19:25:30.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.251+0000 7fe5f2dbc640 10 monclient: init 2024-03-25T19:25:30.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:30.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fe5ec0a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fe5ec13f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fe5ec1430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec142520 con 0x7fe5ec13f710 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec0b28a0 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec0b2a30 con 0x7fe5ec1430c0 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:30.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.252+0000 7fe5f2dbc640 10 monclient(hunting): authenticate will time out at 6657.350670s 2024-03-25T19:25:30.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.261+0000 7fe5f0b58640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe5ec0a98f0 0x7fe5ec0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:48020/0 (socket says 172.21.15.60:48020) 2024-03-25T19:25:30.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.261+0000 7fe5f0b58640 1 -- 172.21.15.60:0/2129576905 learned_addr learned my addr 172.21.15.60:0/2129576905 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:30.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.278+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046068336 0 0) 0x7fe5ec0b28a0 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.278+0000 7fe5eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:25:30.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.278+0000 7fe5eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:30.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.278+0000 7fe5eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:30.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.278+0000 7fe5eb7fe640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:25:30.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.278+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5dc003520 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.294+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417811135 0 0) 0x7fe5ec0b2a30 con 0x7fe5ec1430c0 2024-03-25T19:25:30.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.294+0000 7fe5eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:25:30.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.294+0000 7fe5eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:30.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.294+0000 7fe5eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:30.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.294+0000 7fe5eb7fe640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:25:30.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.294+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec0b28a0 con 0x7fe5ec1430c0 2024-03-25T19:25:30.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.296+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370797027 0 0) 0x7fe5ec142520 con 0x7fe5ec13f710 2024-03-25T19:25:30.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.296+0000 7fe5eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.296+0000 7fe5eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.296+0000 7fe5eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.296+0000 7fe5eb7fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.296+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec0b2a30 con 0x7fe5ec13f710 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.296+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2033038694 0 0) 0x7fe5dc003520 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 >> v1:172.21.15.60:6790/0 conn(0x7fe5ec1430c0 legacy=0x7fe5ec14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 >> v1:172.21.15.60:6789/0 conn(0x7fe5ec13f710 legacy=0x7fe5ec1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 1 monclient: found mon.noname-c 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec0b44c0 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:25:30.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5e0001690 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5eb7fe640 10 monclient: dump: 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:30.298 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:30.299 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:30.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5f2dbc640 5 monclient: authenticate success, global_id 4447 2024-03-25T19:25:30.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.297+0000 7fe5f2dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:30.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5e00017f0 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5eb7fe640 1 -- 172.21.15.60:0/2129576905 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5e0001c00 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5eb7fe640 10 monclient: dump: 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:30.316 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:30.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5f2dbc640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:30.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5f2dbc640 4 set_mon_vals no callback set 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5f2dbc640 10 monclient: shutdown 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5f2dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.316+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/2129576905 >> v1:172.21.15.90:6789/0 conn(0x7fe5ec0a98f0 legacy=0x7fe5ec0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.317+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/2129576905 shutdown_connections 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.317+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/2129576905 >> 172.21.15.60:0/2129576905 conn(0x7fe5ec0b7ea0 msgr2=0x7fe5ec0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.317+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/2129576905 shutdown_connections 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.317+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/2129576905 wait complete. 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 10 monclient: build_initial_monmap 2024-03-25T19:25:30.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 10 monclient: monmap: 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 1 librados: starting msgr at 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 1 librados: starting objecter 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 1 Processor -- start 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 1 -- start start 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 1 librados: setting wanted keys 2024-03-25T19:25:30.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 1 librados: calling monclient init 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.318+0000 7fe5f2dbc640 10 monclient: init 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fe5ec0a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fe5ec13f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fe5ec1430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec139610 con 0x7fe5ec0a98f0 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec139900 con 0x7fe5ec1430c0 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5ec139a90 con 0x7fe5ec13f710 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:30.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.319+0000 7fe5f2dbc640 10 monclient(hunting): authenticate will time out at 6657.417356s 2024-03-25T19:25:30.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.323+0000 7fe5f1359640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe5ec1430c0 0x7fe5ec133260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:48062/0 (socket says 172.21.15.60:48062) 2024-03-25T19:25:30.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.323+0000 7fe5f1359640 1 -- 172.21.15.60:0/230955079 learned_addr learned my addr 172.21.15.60:0/230955079 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:30.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.326+0000 7fe5e97fa640 10 client.?.objecter ms_handle_connect 0x7fe5ec1430c0 2024-03-25T19:25:30.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.326+0000 7fe5e97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:30.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.338+0000 7fe5e97fa640 10 client.?.objecter ms_handle_connect 0x7fe5ec13f710 2024-03-25T19:25:30.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.338+0000 7fe5e97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:30.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.340+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593003817 0 0) 0x7fe5ec139900 con 0x7fe5ec1430c0 2024-03-25T19:25:30.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.340+0000 7fe5e97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:25:30.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.340+0000 7fe5e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:30.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.340+0000 7fe5e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:30.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.340+0000 7fe5e97fa640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:25:30.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.340+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d0004eb0 con 0x7fe5ec1430c0 2024-03-25T19:25:30.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.346+0000 7fe5e97fa640 10 client.?.objecter ms_handle_connect 0x7fe5ec0a98f0 2024-03-25T19:25:30.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.346+0000 7fe5e97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:30.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.353+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 890036726 0 0) 0x7fe5d0004eb0 con 0x7fe5ec1430c0 2024-03-25T19:25:30.352 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.353+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5ec139900 con 0x7fe5ec1430c0 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.353+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5e4001c90 con 0x7fe5ec1430c0 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.353+0000 7fe5e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.353+0000 7fe5e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.353+0000 7fe5e97fa640 10 monclient(hunting): dump: 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:30.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.353+0000 7fe5e97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:30.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.356+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53303740 0 0) 0x7fe5ec139a90 con 0x7fe5ec13f710 2024-03-25T19:25:30.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.356+0000 7fe5e97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:25:30.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.356+0000 7fe5e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:30.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.356+0000 7fe5e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:30.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.356+0000 7fe5e97fa640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:25:30.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.356+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5d0004eb0 con 0x7fe5ec13f710 2024-03-25T19:25:30.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3795251172 0 0) 0x7fe5ec139900 con 0x7fe5ec1430c0 2024-03-25T19:25:30.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 >> v1:172.21.15.60:6790/0 conn(0x7fe5ec13f710 legacy=0x7fe5ec132800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:30.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 >> v1:172.21.15.60:6789/0 conn(0x7fe5ec0a98f0 legacy=0x7fe5ec1320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 1 monclient: found mon.b 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec138bf0 con 0x7fe5ec1430c0 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 10 monclient: _finish_auth 0 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 10 monclient: _renew_subs 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec138e40 con 0x7fe5ec1430c0 2024-03-25T19:25:30.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:30.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 10 monclient: _renew_subs 2024-03-25T19:25:30.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:30.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec138380 con 0x7fe5ec1430c0 2024-03-25T19:25:30.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.363+0000 7fe5f2dbc640 1 librados: init done 2024-03-25T19:25:30.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.364+0000 7fe5f2dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:30.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.364+0000 7fe5f2dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:30.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.364+0000 7fe5f2dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:30.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.364+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ec0627b0 con 0x7fe5ec1430c0 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.668+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5e4003970 con 0x7fe5ec1430c0 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.668+0000 7fe5e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.668+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe5e4003d00 con 0x7fe5ec1430c0 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.668+0000 7fe5e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.668+0000 7fe5e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.668+0000 7fe5e97fa640 10 monclient: dump: 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:30.668 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:30.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.668+0000 7fe5eaffd640 4 set_mon_vals no callback set 2024-03-25T19:25:30.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.669+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fe5e4003f20 con 0x7fe5ec1430c0 2024-03-25T19:25:30.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.670+0000 7fe5f0b58640 10 monclient: get_auth_request con 0x7fe5d006dd90 auth_method 0 2024-03-25T19:25:30.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.670+0000 7fe5e97fa640 10 client.4450.objecter ms_handle_connect 0x7fe5d006dd90 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fe5e4054aa0 con 0x7fe5ec1430c0 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 10 client.4450.objecter ms_dispatch 0x7fe5ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5e40505b0 con 0x7fe5ec1430c0 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:30.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.675+0000 7fe5e97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:30.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.907+0000 7fe5f2dbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:30.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.907+0000 7fe5f2dbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:30.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.907+0000 7fe5f2dbc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:30.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:30.907+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5ec141a00 con 0x7fe5ec1430c0 2024-03-25T19:25:31.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.052+0000 7fe5e97fa640 1 -- 172.21.15.60:0/230955079 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fe5e40524e0 con 0x7fe5ec1430c0 2024-03-25T19:25:31.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.053+0000 7fe5e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:31.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.053+0000 7fe5e97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:31.053 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:31.053 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:31.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 10 librados: watch_flush enter 2024-03-25T19:25:31.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 10 librados: watch_flush exit 2024-03-25T19:25:31.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T19:25:31.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 10 client.4450.objecter successfully canceled tick 2024-03-25T19:25:31.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 >> v1:172.21.15.90:6800/1827638754 conn(0x7fe5d006dd90 legacy=0x7fe5d0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 10 monclient: shutdown 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 >> v1:172.21.15.90:6789/0 conn(0x7fe5ec1430c0 legacy=0x7fe5ec133260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 shutdown_connections 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 >> 172.21.15.60:0/230955079 conn(0x7fe5ec0b7ea0 msgr2=0x7fe5ec074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 shutdown_connections 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 1 -- 172.21.15.60:0/230955079 wait complete. 2024-03-25T19:25:31.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:31.055+0000 7fe5f2dbc640 1 librados: shutdown 2024-03-25T19:25:31.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:34.638 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:05:35.349 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.647 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:05:35.358 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.688 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:35.342 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.700 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:35.354 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.708 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:05:35.419 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.709 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:05:35.421 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.715 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:05:35.426 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.727 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:05:35.381 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:34.738 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:05:35.449 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:37.873 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.990+0000 7f4d97755640 10 monclient: get_monmap_and_config 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.990+0000 7f4d97755640 10 monclient: build_initial_monmap 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.990+0000 7f4d97755640 10 monclient: monmap: 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:25:37.991323+0000 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:25:37.991323+0000 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:25:37.990 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:25:37.991 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.991+0000 7f4d97755640 1 Processor -- start 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.991+0000 7f4d97755640 1 -- start start 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.991+0000 7f4d97755640 10 monclient: init 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 10 monclient(hunting): picked mon.noname-a con 0x7f4d900b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 10 monclient(hunting): picked mon.noname-b con 0x7f4d900b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 10 monclient(hunting): picked mon.noname-c con 0x7f4d9014be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d900af940 con 0x7f4d900b4d60 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d900b0a70 con 0x7f4d9014be90 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d900b0d50 con 0x7f4d900b1730 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:37.992 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.992+0000 7f4d97755640 10 monclient(hunting): authenticate will time out at 6665.090018s 2024-03-25T19:25:37.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.999+0000 7f4d94cf0640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f4d900b1730 0x7f4d900b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:48490/0 (socket says 172.21.15.60:48490) 2024-03-25T19:25:37.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:37.999+0000 7f4d94cf0640 1 -- 172.21.15.60:0/3227475815 learned_addr learned my addr 172.21.15.60:0/3227475815 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:38.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.018+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796567994 0 0) 0x7f4d900b0d50 con 0x7f4d900b1730 2024-03-25T19:25:38.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.019+0000 7f4d87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:38.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.019+0000 7f4d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:38.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.019+0000 7f4d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:38.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.019+0000 7f4d87fff640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:25:38.018 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.019+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d7c003520 con 0x7f4d900b1730 2024-03-25T19:25:38.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.025+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448308262 0 0) 0x7f4d900b0a70 con 0x7f4d9014be90 2024-03-25T19:25:38.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.026+0000 7f4d87fff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:38.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.026+0000 7f4d87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:38.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.026+0000 7f4d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:38.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.026+0000 7f4d87fff640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:25:38.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.026+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d900b0d50 con 0x7f4d9014be90 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1683532716 0 0) 0x7f4d7c003520 con 0x7f4d900b1730 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 >> v1:172.21.15.90:6789/0 conn(0x7f4d9014be90 legacy=0x7f4d9014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 >> v1:172.21.15.60:6789/0 conn(0x7f4d900b4d60 legacy=0x7f4d9013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 1 monclient: found mon.noname-b 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d900b1030 con 0x7f4d900b1730 2024-03-25T19:25:38.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 10 monclient: _finish_auth 0 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4d80001690 con 0x7f4d900b1730 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d87fff640 10 monclient: dump: 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:38.031 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:38.032 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:38.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d97755640 5 monclient: authenticate success, global_id 4448 2024-03-25T19:25:38.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.030+0000 7f4d97755640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d800017f0 con 0x7f4d900b1730 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d87fff640 1 -- 172.21.15.60:0/3227475815 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4d80001c00 con 0x7f4d900b1730 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d87fff640 10 monclient: dump: 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:38.050 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d97755640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:38.051 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.050+0000 7f4d97755640 4 set_mon_vals no callback set 2024-03-25T19:25:38.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.051+0000 7f4d97755640 10 monclient: shutdown 2024-03-25T19:25:38.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.051+0000 7f4d97755640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:38.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.051+0000 7f4d97755640 1 -- 172.21.15.60:0/3227475815 >> v1:172.21.15.60:6790/0 conn(0x7f4d900b1730 legacy=0x7f4d900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:38.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.051+0000 7f4d97755640 1 -- 172.21.15.60:0/3227475815 shutdown_connections 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.051+0000 7f4d97755640 1 -- 172.21.15.60:0/3227475815 >> 172.21.15.60:0/3227475815 conn(0x7f4d900617a0 msgr2=0x7f4d90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.052+0000 7f4d97755640 1 -- 172.21.15.60:0/3227475815 shutdown_connections 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.052+0000 7f4d97755640 1 -- 172.21.15.60:0/3227475815 wait complete. 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.052+0000 7f4d97755640 10 monclient: build_initial_monmap 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.052+0000 7f4d97755640 10 monclient: monmap: 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:25:38.054 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.053+0000 7f4d97755640 1 librados: starting msgr at 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.053+0000 7f4d97755640 1 librados: starting objecter 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.053+0000 7f4d97755640 1 Processor -- start 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.053+0000 7f4d97755640 1 -- start start 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.053+0000 7f4d97755640 1 librados: setting wanted keys 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.053+0000 7f4d97755640 1 librados: calling monclient init 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.053+0000 7f4d97755640 10 monclient: init 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 10 monclient(hunting): picked mon.noname-c con 0x7f4d900b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 10 monclient(hunting): picked mon.noname-b con 0x7f4d900b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 10 monclient(hunting): picked mon.noname-a con 0x7f4d9014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d90144330 con 0x7f4d9014be90 2024-03-25T19:25:38.055 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d90136b20 con 0x7f4d900b4d60 2024-03-25T19:25:38.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d901448b0 con 0x7f4d900b1730 2024-03-25T19:25:38.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:38.056 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.054+0000 7f4d97755640 10 monclient(hunting): authenticate will time out at 6665.152029s 2024-03-25T19:25:38.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.065+0000 7f4d94cf0640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4d900b4d60 0x7f4d90137220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:38780/0 (socket says 172.21.15.60:38780) 2024-03-25T19:25:38.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.065+0000 7f4d94cf0640 1 -- 172.21.15.60:0/3664906069 learned_addr learned my addr 172.21.15.60:0/3664906069 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:38.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.065+0000 7f4d85ffb640 10 client.?.objecter ms_handle_connect 0x7f4d900b4d60 2024-03-25T19:25:38.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.065+0000 7f4d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:38.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.069+0000 7f4d85ffb640 10 client.?.objecter ms_handle_connect 0x7f4d900b1730 2024-03-25T19:25:38.068 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.069+0000 7f4d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:38.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226699722 0 0) 0x7f4d90136b20 con 0x7f4d900b4d60 2024-03-25T19:25:38.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:38.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:38.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:38.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:25:38.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d74004eb0 con 0x7f4d900b4d60 2024-03-25T19:25:38.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 10 client.?.objecter ms_handle_connect 0x7f4d9014be90 2024-03-25T19:25:38.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.075+0000 7f4d85ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:38.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.082+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856907387 0 0) 0x7f4d90144330 con 0x7f4d9014be90 2024-03-25T19:25:38.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.082+0000 7f4d85ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:38.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.082+0000 7f4d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:38.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.082+0000 7f4d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:38.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.082+0000 7f4d85ffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:25:38.082 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.082+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d90136b20 con 0x7f4d9014be90 2024-03-25T19:25:38.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.084+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491140249 0 0) 0x7f4d901448b0 con 0x7f4d900b1730 2024-03-25T19:25:38.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.084+0000 7f4d85ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:38.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.084+0000 7f4d85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:38.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.084+0000 7f4d85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:38.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.084+0000 7f4d85ffb640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:25:38.083 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.084+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d90144330 con 0x7f4d900b1730 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.087+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3310090041 0 0) 0x7f4d74004eb0 con 0x7f4d900b4d60 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.087+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d901448b0 con 0x7f4d900b4d60 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.087+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4d80001c00 con 0x7f4d900b4d60 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.087+0000 7f4d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.087+0000 7f4d85ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.087+0000 7f4d85ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:38.087 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:38.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.087+0000 7f4d85ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.096+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 660999610 0 0) 0x7f4d90136b20 con 0x7f4d9014be90 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.096+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d74004eb0 con 0x7f4d9014be90 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.096+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4d88001b90 con 0x7f4d9014be90 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.096+0000 7f4d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.096+0000 7f4d85ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.096+0000 7f4d85ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:38.096 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:38.097 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:38.097 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:38.097 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:38.097 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.106+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1049900251 0 0) 0x7f4d90144330 con 0x7f4d900b1730 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.106+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d90136b20 con 0x7f4d900b1730 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.106+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4d8c001860 con 0x7f4d900b1730 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.106+0000 7f4d85ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.106+0000 7f4d85ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.106+0000 7f4d85ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.106 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:38.107 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:38.107 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:38.107 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:38.107 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:38.107 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2764389351 0 0) 0x7f4d901448b0 con 0x7f4d900b4d60 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 >> v1:172.21.15.60:6790/0 conn(0x7f4d900b1730 legacy=0x7f4d9013ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 >> v1:172.21.15.60:6789/0 conn(0x7f4d9014be90 legacy=0x7f4d901e24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 1 monclient: found mon.b 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:38.108 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d90144da0 con 0x7f4d900b4d60 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 10 monclient: _renew_subs 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d90145050 con 0x7f4d900b4d60 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 10 monclient: _renew_subs 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d90135ce0 con 0x7f4d900b4d60 2024-03-25T19:25:38.109 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.108+0000 7f4d97755640 1 librados: init done 2024-03-25T19:25:38.110 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.110+0000 7f4d97755640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:38.110 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.110+0000 7f4d97755640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:38.110 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.110+0000 7f4d97755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:38.110 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.110+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d900627b0 con 0x7f4d900b4d60 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.382+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d80004a90 con 0x7f4d900b4d60 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.382+0000 7f4d85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.382+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4d80004e20 con 0x7f4d900b4d60 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.382+0000 7f4d85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.382+0000 7f4d85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.382+0000 7f4d85ffb640 10 monclient: dump: 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:38.382 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:38.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.382+0000 7f4d877fe640 4 set_mon_vals no callback set 2024-03-25T19:25:38.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.383+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f4d80005000 con 0x7f4d900b4d60 2024-03-25T19:25:38.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.384+0000 7f4d954f1640 10 monclient: get_auth_request con 0x7f4d7406e0b0 auth_method 0 2024-03-25T19:25:38.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.384+0000 7f4d85ffb640 10 client.4456.objecter ms_handle_connect 0x7f4d7406e0b0 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f4d80054a50 con 0x7f4d900b4d60 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 10 client.4456.objecter ms_dispatch 0x7f4d90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d80050560 con 0x7f4d900b4d60 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:38.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.389+0000 7f4d85ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:38.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.644+0000 7f4d97755640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:38.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.644+0000 7f4d97755640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:38.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.644+0000 7f4d97755640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:38.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.644+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d90069f40 con 0x7f4d900b4d60 2024-03-25T19:25:38.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.670+0000 7f4d85ffb640 1 -- 172.21.15.60:0/3664906069 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f4d80052490 con 0x7f4d900b4d60 2024-03-25T19:25:38.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.670+0000 7f4d85ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:38.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.670+0000 7f4d85ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:38.670 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:38.670 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:38.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 10 librados: watch_flush enter 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 10 librados: watch_flush exit 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 10 client.4456.objecter successfully canceled tick 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 >> v1:172.21.15.90:6800/1827638754 conn(0x7f4d7406e0b0 legacy=0x7f4d74070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 10 monclient: shutdown 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 >> v1:172.21.15.90:6789/0 conn(0x7f4d900b4d60 legacy=0x7f4d90137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 shutdown_connections 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.672+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 >> 172.21.15.60:0/3664906069 conn(0x7f4d900617a0 msgr2=0x7f4d900b6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.673+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 shutdown_connections 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.673+0000 7f4d97755640 1 -- 172.21.15.60:0/3664906069 wait complete. 2024-03-25T19:25:38.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:38.673+0000 7f4d97755640 1 librados: shutdown 2024-03-25T19:25:38.685 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:45.491 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.615+0000 7fdb4275d640 10 monclient: get_monmap_and_config 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.615+0000 7fdb4275d640 10 monclient: build_initial_monmap 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.615+0000 7fdb4275d640 10 monclient: monmap: 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:25:45.616341+0000 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:25:45.616341+0000 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:45.615 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:45.616 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:25:45.616 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:25:45.616 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:45.616 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.616+0000 7fdb4275d640 1 Processor -- start 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.616+0000 7fdb4275d640 1 -- start start 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.616+0000 7fdb4275d640 10 monclient: init 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3c14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0af940 con 0x7fdb3c0b1730 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0b0a70 con 0x7fdb3c14be90 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c0b0d50 con 0x7fdb3c0b4d60 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:45.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.617+0000 7fdb4275d640 10 monclient(hunting): authenticate will time out at 6672.715220s 2024-03-25T19:25:45.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.620+0000 7fdb40cfa640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fdb3c14be90 0x7fdb3c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:43462/0 (socket says 172.21.15.60:43462) 2024-03-25T19:25:45.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.620+0000 7fdb40cfa640 1 -- 172.21.15.60:0/3216970963 learned_addr learned my addr 172.21.15.60:0/3216970963 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:45.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.637+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206933384 0 0) 0x7fdb3c0b0a70 con 0x7fdb3c14be90 2024-03-25T19:25:45.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.637+0000 7fdb3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:45.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.637+0000 7fdb3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:45.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.637+0000 7fdb3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:45.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.637+0000 7fdb3affd640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:25:45.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.637+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb24003520 con 0x7fdb3c14be90 2024-03-25T19:25:45.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.643+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813739374 0 0) 0x7fdb3c0af940 con 0x7fdb3c0b1730 2024-03-25T19:25:45.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.643+0000 7fdb3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:45.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.643+0000 7fdb3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:45.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.643+0000 7fdb3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:45.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.643+0000 7fdb3affd640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:25:45.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.643+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c0b0a70 con 0x7fdb3c0b1730 2024-03-25T19:25:45.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.654+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571433961 0 0) 0x7fdb3c0b0d50 con 0x7fdb3c0b4d60 2024-03-25T19:25:45.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.655+0000 7fdb3affd640 10 monclient(hunting): _negotiate 2024-03-25T19:25:45.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.655+0000 7fdb3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:45.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.655+0000 7fdb3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:45.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.655+0000 7fdb3affd640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:25:45.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.655+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c0af940 con 0x7fdb3c0b4d60 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2272032714 0 0) 0x7fdb3c0b0a70 con 0x7fdb3c0b1730 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 >> v1:172.21.15.60:6790/0 conn(0x7fdb3c0b4d60 legacy=0x7fdb3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 >> v1:172.21.15.90:6789/0 conn(0x7fdb3c14be90 legacy=0x7fdb3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 10 monclient: _finish_hunting 0 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 1 monclient: found mon.noname-a 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c0b1030 con 0x7fdb3c0b1730 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 10 monclient: _finish_auth 0 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 10 monclient: _check_auth_tickets 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:45.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdb30001670 con 0x7fdb3c0b1730 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb3affd640 10 monclient: dump: 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:45.662 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:45.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb4275d640 5 monclient: authenticate success, global_id 4470 2024-03-25T19:25:45.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.661+0000 7fdb4275d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:45.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb300017d0 con 0x7fdb3c0b1730 2024-03-25T19:25:45.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:45.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb3affd640 1 -- 172.21.15.60:0/3216970963 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdb30001be0 con 0x7fdb3c0b1730 2024-03-25T19:25:45.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:45.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:45.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb3affd640 10 monclient: dump: 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:45.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 4 set_mon_vals no callback set 2024-03-25T19:25:45.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 10 monclient: shutdown 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 1 -- 172.21.15.60:0/3216970963 >> v1:172.21.15.60:6789/0 conn(0x7fdb3c0b1730 legacy=0x7fdb3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 1 -- 172.21.15.60:0/3216970963 shutdown_connections 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 1 -- 172.21.15.60:0/3216970963 >> 172.21.15.60:0/3216970963 conn(0x7fdb3c0617a0 msgr2=0x7fdb3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 1 -- 172.21.15.60:0/3216970963 shutdown_connections 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.673+0000 7fdb4275d640 1 -- 172.21.15.60:0/3216970963 wait complete. 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.674+0000 7fdb4275d640 10 monclient: build_initial_monmap 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.674+0000 7fdb4275d640 10 monclient: monmap: 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:25:45.676 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.674+0000 7fdb4275d640 1 librados: starting msgr at 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.674+0000 7fdb4275d640 1 librados: starting objecter 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.675+0000 7fdb4275d640 1 Processor -- start 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.675+0000 7fdb4275d640 1 -- start start 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.675+0000 7fdb4275d640 1 librados: setting wanted keys 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.675+0000 7fdb4275d640 1 librados: calling monclient init 2024-03-25T19:25:45.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.675+0000 7fdb4275d640 10 monclient: init 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.675+0000 7fdb4275d640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.675+0000 7fdb4275d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3c0b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c13aa80 con 0x7fdb3c14be90 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c137c60 con 0x7fdb3c0b1730 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb3c1326c0 con 0x7fdb3c0b4d60 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:45.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.676+0000 7fdb4275d640 10 monclient(hunting): authenticate will time out at 6672.774025s 2024-03-25T19:25:45.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.681+0000 7fdb40cfa640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fdb3c14be90 0x7fdb3c1d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:51632/0 (socket says 172.21.15.60:51632) 2024-03-25T19:25:45.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.681+0000 7fdb40cfa640 1 -- 172.21.15.60:0/3651675818 learned_addr learned my addr 172.21.15.60:0/3651675818 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:45.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.681+0000 7fdb38ff9640 10 client.?.objecter ms_handle_connect 0x7fdb3c14be90 2024-03-25T19:25:45.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.681+0000 7fdb38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:45.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.685+0000 7fdb38ff9640 10 client.?.objecter ms_handle_connect 0x7fdb3c0b1730 2024-03-25T19:25:45.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.685+0000 7fdb38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:45.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869113593 0 0) 0x7fdb3c13aa80 con 0x7fdb3c14be90 2024-03-25T19:25:45.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:45.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:45.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb28004eb0 con 0x7fdb3c14be90 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 client.?.objecter ms_handle_connect 0x7fdb3c0b4d60 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639949350 0 0) 0x7fdb3c137c60 con 0x7fdb3c0b1730 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:45.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:25:45.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.698+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c13aa80 con 0x7fdb3c0b1730 2024-03-25T19:25:45.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427719881 0 0) 0x7fdb3c1326c0 con 0x7fdb3c0b4d60 2024-03-25T19:25:45.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:25:45.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:45.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:45.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:25:45.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c137c60 con 0x7fdb3c0b4d60 2024-03-25T19:25:45.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961252703 0 0) 0x7fdb3c13aa80 con 0x7fdb3c0b1730 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb3c1326c0 con 0x7fdb3c0b1730 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdb2c001870 con 0x7fdb3c0b1730 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.712+0000 7fdb38ff9640 10 monclient(hunting): dump: 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:45.713 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:45.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.713+0000 7fdb38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.733+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 446300909 0 0) 0x7fdb28004eb0 con 0x7fdb3c14be90 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.733+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb3c13aa80 con 0x7fdb3c14be90 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdb34001b30 con 0x7fdb3c14be90 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 10 monclient(hunting): dump: 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:45.734 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 959525288 0 0) 0x7fdb3c1326c0 con 0x7fdb3c0b1730 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 >> v1:172.21.15.60:6790/0 conn(0x7fdb3c0b4d60 legacy=0x7fdb3c1d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 >> v1:172.21.15.60:6789/0 conn(0x7fdb3c14be90 legacy=0x7fdb3c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 1 monclient: found mon.b 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3c138440 con 0x7fdb3c0b1730 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 10 monclient: _finish_auth 0 2024-03-25T19:25:45.735 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 5 monclient: authenticate success, global_id 4462 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 10 monclient: _renew_subs 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb3c138e60 con 0x7fdb3c0b1730 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 10 monclient: _renew_subs 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3c145260 con 0x7fdb3c0b1730 2024-03-25T19:25:45.736 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.734+0000 7fdb4275d640 1 librados: init done 2024-03-25T19:25:45.737 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.735+0000 7fdb4275d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:45.737 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.735+0000 7fdb4275d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:45.737 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.735+0000 7fdb4275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:45.737 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.735+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c0627b0 con 0x7fdb3c0b1730 2024-03-25T19:25:45.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb2c003610 con 0x7fdb3c0b1730 2024-03-25T19:25:45.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:45.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fdb2c003a60 con 0x7fdb3c0b1730 2024-03-25T19:25:45.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:45.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb38ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:45.759 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb38ff9640 10 monclient: dump: 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:45.760 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb3a7fc640 4 set_mon_vals no callback set 2024-03-25T19:25:45.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.760+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fdb2c003c80 con 0x7fdb3c0b1730 2024-03-25T19:25:45.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.761+0000 7fdb3b7fe640 10 monclient: get_auth_request con 0x7fdb2806df60 auth_method 0 2024-03-25T19:25:45.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.761+0000 7fdb38ff9640 10 client.4462.objecter ms_handle_connect 0x7fdb2806df60 2024-03-25T19:25:45.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.767+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fdb2c0535d0 con 0x7fdb3c0b1730 2024-03-25T19:25:45.766 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.767+0000 7fdb38ff9640 10 client.4462.objecter ms_dispatch 0x7fdb3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:45.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.767+0000 7fdb38ff9640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:45.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.767+0000 7fdb38ff9640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:45.767 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:45.767+0000 7fdb38ff9640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T19:25:46.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.028+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb2c04f0e0 con 0x7fdb3c0b1730 2024-03-25T19:25:46.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.028+0000 7fdb38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:46.028 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.028+0000 7fdb38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:46.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.283+0000 7fdb4275d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:46.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.283+0000 7fdb4275d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:46.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.283+0000 7fdb4275d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:46.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.283+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb3c1385d0 con 0x7fdb3c0b1730 2024-03-25T19:25:46.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.314+0000 7fdb38ff9640 1 -- 172.21.15.60:0/3651675818 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fdb2c051010 con 0x7fdb3c0b1730 2024-03-25T19:25:46.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.314+0000 7fdb38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:46.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.314+0000 7fdb38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:46.314 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:46.314 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:46.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 10 librados: watch_flush enter 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 10 librados: watch_flush exit 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 10 client.4462.objecter successfully canceled tick 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 >> v1:172.21.15.90:6800/1827638754 conn(0x7fdb2806df60 legacy=0x7fdb28070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 10 monclient: shutdown 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.316+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 >> v1:172.21.15.90:6789/0 conn(0x7fdb3c0b1730 legacy=0x7fdb3c131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.317+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 shutdown_connections 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.317+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 >> 172.21.15.60:0/3651675818 conn(0x7fdb3c0617a0 msgr2=0x7fdb3c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.317+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 shutdown_connections 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.317+0000 7fdb4275d640 1 -- 172.21.15.60:0/3651675818 wait complete. 2024-03-25T19:25:46.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:46.317+0000 7fdb4275d640 1 librados: shutdown 2024-03-25T19:25:46.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:53.135 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:25:53.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.254+0000 7fd9db5ba640 10 monclient: get_monmap_and_config 2024-03-25T19:25:53.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.254+0000 7fd9db5ba640 10 monclient: build_initial_monmap 2024-03-25T19:25:53.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.254+0000 7fd9db5ba640 10 monclient: monmap: 2024-03-25T19:25:53.253 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:25:53.254881+0000 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:25:53.254881+0000 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:25:53.254 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.254+0000 7fd9db5ba640 1 Processor -- start 2024-03-25T19:25:53.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 1 -- start start 2024-03-25T19:25:53.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient: init 2024-03-25T19:25:53.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:53.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d40b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d40b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d414be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40af940 con 0x7fd9d40b4d60 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40b0a70 con 0x7fd9d40b1730 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40b0d50 con 0x7fd9d414be90 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:53.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.255+0000 7fd9db5ba640 10 monclient(hunting): authenticate will time out at 6680.353628s 2024-03-25T19:25:53.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.270+0000 7fd9d8b55640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd9d40b1730 0x7fd9d40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:39380/0 (socket says 172.21.15.60:39380) 2024-03-25T19:25:53.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.270+0000 7fd9d8b55640 1 -- 172.21.15.60:0/2111411999 learned_addr learned my addr 172.21.15.60:0/2111411999 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:53.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.283+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184992920 0 0) 0x7fd9d40b0a70 con 0x7fd9d40b1730 2024-03-25T19:25:53.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.284+0000 7fd9cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:53.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.284+0000 7fd9cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:53.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.284+0000 7fd9cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:53.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.284+0000 7fd9cbfff640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:25:53.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.284+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0003520 con 0x7fd9d40b1730 2024-03-25T19:25:53.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.294+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293072163 0 0) 0x7fd9d40af940 con 0x7fd9d40b4d60 2024-03-25T19:25:53.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.294+0000 7fd9cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:25:53.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.294+0000 7fd9cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:53.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.294+0000 7fd9cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:53.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.294+0000 7fd9cbfff640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:25:53.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.294+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d40b0a70 con 0x7fd9d40b4d60 2024-03-25T19:25:53.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911701147 0 0) 0x7fd9c0003520 con 0x7fd9d40b1730 2024-03-25T19:25:53.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 >> v1:172.21.15.60:6790/0 conn(0x7fd9d414be90 legacy=0x7fd9d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:53.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 >> v1:172.21.15.60:6789/0 conn(0x7fd9d40b4d60 legacy=0x7fd9d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:53.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 1 monclient: found mon.noname-c 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40b1030 con 0x7fd9d40b1730 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 10 monclient: _finish_auth 0 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd9c4001690 con 0x7fd9d40b1730 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9db5ba640 5 monclient: authenticate success, global_id 4465 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9db5ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9cbfff640 10 monclient: dump: 2024-03-25T19:25:53.299 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.298+0000 7fd9db5ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:25:53.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c40017f0 con 0x7fd9d40b1730 2024-03-25T19:25:53.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:53.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9cbfff640 1 -- 172.21.15.60:0/2111411999 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd9c4001c00 con 0x7fd9d40b1730 2024-03-25T19:25:53.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:53.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9cbfff640 10 monclient: dump: 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9db5ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:25:53.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9db5ba640 4 set_mon_vals no callback set 2024-03-25T19:25:53.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9db5ba640 10 monclient: shutdown 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9db5ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9db5ba640 1 -- 172.21.15.60:0/2111411999 >> v1:172.21.15.90:6789/0 conn(0x7fd9d40b1730 legacy=0x7fd9d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9db5ba640 1 -- 172.21.15.60:0/2111411999 shutdown_connections 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.319+0000 7fd9db5ba640 1 -- 172.21.15.60:0/2111411999 >> 172.21.15.60:0/2111411999 conn(0x7fd9d40617a0 msgr2=0x7fd9d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.320+0000 7fd9db5ba640 1 -- 172.21.15.60:0/2111411999 shutdown_connections 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.320+0000 7fd9db5ba640 1 -- 172.21.15.60:0/2111411999 wait complete. 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.320+0000 7fd9db5ba640 10 monclient: build_initial_monmap 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.320+0000 7fd9db5ba640 10 monclient: monmap: 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:25:53.321 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.321+0000 7fd9db5ba640 1 librados: starting msgr at 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.321+0000 7fd9db5ba640 1 librados: starting objecter 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.321+0000 7fd9db5ba640 1 Processor -- start 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.321+0000 7fd9db5ba640 1 -- start start 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.321+0000 7fd9db5ba640 1 librados: setting wanted keys 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.321+0000 7fd9db5ba640 1 librados: calling monclient init 2024-03-25T19:25:53.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.321+0000 7fd9db5ba640 10 monclient: init 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d40b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d40b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4134990 con 0x7fd9d414be90 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4141b50 con 0x7fd9d40b4d60 2024-03-25T19:25:53.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4141e30 con 0x7fd9d40b1730 2024-03-25T19:25:53.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:25:53.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.322+0000 7fd9db5ba640 10 monclient(hunting): authenticate will time out at 6680.420155s 2024-03-25T19:25:53.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.331+0000 7fd9d9b57640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd9d414be90 0x7fd9d4134280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:56156/0 (socket says 172.21.15.60:56156) 2024-03-25T19:25:53.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.331+0000 7fd9d9b57640 1 -- 172.21.15.60:0/3176548433 learned_addr learned my addr 172.21.15.60:0/3176548433 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:25:53.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.334+0000 7fd9c9ffb640 10 client.?.objecter ms_handle_connect 0x7fd9d414be90 2024-03-25T19:25:53.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.334+0000 7fd9c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:53.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.338+0000 7fd9c9ffb640 10 client.?.objecter ms_handle_connect 0x7fd9d40b4d60 2024-03-25T19:25:53.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.338+0000 7fd9c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:53.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.340+0000 7fd9c9ffb640 10 client.?.objecter ms_handle_connect 0x7fd9d40b1730 2024-03-25T19:25:53.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.340+0000 7fd9c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:25:53.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.345+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230590114 0 0) 0x7fd9d4141e30 con 0x7fd9d40b1730 2024-03-25T19:25:53.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.345+0000 7fd9c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:53.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.345+0000 7fd9c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:53.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.345+0000 7fd9c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:53.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.345+0000 7fd9c9ffb640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:25:53.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.345+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8004eb0 con 0x7fd9d40b1730 2024-03-25T19:25:53.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.349+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167636091 0 0) 0x7fd9d4134990 con 0x7fd9d414be90 2024-03-25T19:25:53.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.349+0000 7fd9c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:53.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.349+0000 7fd9c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:53.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.349+0000 7fd9c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:53.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.349+0000 7fd9c9ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:25:53.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.349+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d4141e30 con 0x7fd9d414be90 2024-03-25T19:25:53.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.355+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890865547 0 0) 0x7fd9d4141b50 con 0x7fd9d40b4d60 2024-03-25T19:25:53.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.355+0000 7fd9c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:25:53.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.355+0000 7fd9c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:25:53.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.355+0000 7fd9c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:25:53.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.355+0000 7fd9c9ffb640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:25:53.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.355+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d4134990 con 0x7fd9d40b4d60 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.365+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1919222008 0 0) 0x7fd9b8004eb0 con 0x7fd9d40b1730 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.365+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d4141b50 con 0x7fd9d40b1730 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.365+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd9d0001930 con 0x7fd9d40b1730 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.365+0000 7fd9c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.365+0000 7fd9c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.365+0000 7fd9c9ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:53.365 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.365+0000 7fd9c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.366+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154455392 0 0) 0x7fd9d4134990 con 0x7fd9d40b4d60 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.366+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9b8004eb0 con 0x7fd9d40b4d60 2024-03-25T19:25:53.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.366+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd9c4001b00 con 0x7fd9d40b4d60 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.366+0000 7fd9c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.366+0000 7fd9c9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.366+0000 7fd9c9ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:53.367 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.370+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149723419 0 0) 0x7fd9d4141e30 con 0x7fd9d414be90 2024-03-25T19:25:53.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.370+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d4134990 con 0x7fd9d414be90 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.370+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd9cc001bd0 con 0x7fd9d414be90 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.370+0000 7fd9c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.370+0000 7fd9c9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.370+0000 7fd9c9ffb640 10 monclient(hunting): dump: 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:53.370 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:53.371 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:53.371 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3985665681 0 0) 0x7fd9b8004eb0 con 0x7fd9d40b4d60 2024-03-25T19:25:53.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 >> v1:172.21.15.60:6790/0 conn(0x7fd9d40b1730 legacy=0x7fd9d4142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:53.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 >> v1:172.21.15.60:6789/0 conn(0x7fd9d414be90 legacy=0x7fd9d4134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:53.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:25:53.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 1 monclient: found mon.b 2024-03-25T19:25:53.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d41406d0 con 0x7fd9d40b4d60 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 10 monclient: _renew_subs 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d4140380 con 0x7fd9d40b4d60 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 10 monclient: _renew_subs 2024-03-25T19:25:53.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:53.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d413bb10 con 0x7fd9d40b4d60 2024-03-25T19:25:53.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.376+0000 7fd9db5ba640 1 librados: init done 2024-03-25T19:25:53.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.377+0000 7fd9db5ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:53.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.377+0000 7fd9db5ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:53.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.377+0000 7fd9db5ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:53.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.377+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d413bdc0 con 0x7fd9d40b4d60 2024-03-25T19:25:53.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.655+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c4004a90 con 0x7fd9d40b4d60 2024-03-25T19:25:53.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.655+0000 7fd9c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:25:53.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.655+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd9c4004da0 con 0x7fd9d40b4d60 2024-03-25T19:25:53.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.655+0000 7fd9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:25:53.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.655+0000 7fd9c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.655+0000 7fd9c9ffb640 10 monclient: dump: 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:25:53.656 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:25:53.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.655+0000 7fd9cb7fe640 4 set_mon_vals no callback set 2024-03-25T19:25:53.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.656+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fd9c401ad70 con 0x7fd9d40b4d60 2024-03-25T19:25:53.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.657+0000 7fd9d9356640 10 monclient: get_auth_request con 0x7fd9b806e160 auth_method 0 2024-03-25T19:25:53.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.657+0000 7fd9c9ffb640 10 client.4468.objecter ms_handle_connect 0x7fd9b806e160 2024-03-25T19:25:53.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.662+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fd9c4054860 con 0x7fd9d40b4d60 2024-03-25T19:25:53.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.662+0000 7fd9c9ffb640 10 client.4468.objecter ms_dispatch 0x7fd9d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:25:53.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.662+0000 7fd9c9ffb640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:25:53.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.662+0000 7fd9c9ffb640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:25:53.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.662+0000 7fd9c9ffb640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T19:25:53.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.663+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9c4050370 con 0x7fd9d40b4d60 2024-03-25T19:25:53.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.663+0000 7fd9c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:25:53.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.663+0000 7fd9c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:25:53.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.896+0000 7fd9db5ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:53.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.896+0000 7fd9db5ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:53.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.896+0000 7fd9db5ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:25:53.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.896+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9d4140510 con 0x7fd9d40b4d60 2024-03-25T19:25:53.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.932+0000 7fd9c9ffb640 1 -- 172.21.15.60:0/3176548433 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fd9c40522a0 con 0x7fd9d40b4d60 2024-03-25T19:25:53.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.932+0000 7fd9c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:25:53.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.932+0000 7fd9c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:25:53.931 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:25:53.932 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 10 librados: watch_flush enter 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 10 librados: watch_flush exit 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 10 client.4468.objecter successfully canceled tick 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 >> v1:172.21.15.90:6800/1827638754 conn(0x7fd9b806e160 legacy=0x7fd9b8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 10 monclient: shutdown 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 >> v1:172.21.15.90:6789/0 conn(0x7fd9d40b4d60 legacy=0x7fd9d4133b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 shutdown_connections 2024-03-25T19:25:53.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.934+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 >> 172.21.15.60:0/3176548433 conn(0x7fd9d40617a0 msgr2=0x7fd9d413c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:25:53.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.935+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 shutdown_connections 2024-03-25T19:25:53.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.935+0000 7fd9db5ba640 1 -- 172.21.15.60:0/3176548433 wait complete. 2024-03-25T19:25:53.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:25:53.935+0000 7fd9db5ba640 1 librados: shutdown 2024-03-25T19:25:53.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:25:59.690 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:00.401 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.735 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:00.389 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.739 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:00.393 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.742 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:00.453 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.778 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:00.432 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.792 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:00.504 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.805 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:00.517 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.821 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:00.533 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:25:59.848 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:00.560 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:00.753 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:00.857 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.857+0000 7f73ffd8d640 10 monclient: get_monmap_and_config 2024-03-25T19:26:00.857 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.857+0000 7f73ffd8d640 10 monclient: build_initial_monmap 2024-03-25T19:26:00.857 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.857+0000 7f73ffd8d640 10 monclient: monmap: 2024-03-25T19:26:00.857 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:00.857 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:00.857 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:00.858301+0000 2024-03-25T19:26:00.858 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:00.858301+0000 2024-03-25T19:26:00.858 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:00.858 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:00.858 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:00.858 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:00.858 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:00.858 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:00.859 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.858+0000 7f73ffd8d640 1 Processor -- start 2024-03-25T19:26:00.859 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.858+0000 7f73ffd8d640 1 -- start start 2024-03-25T19:26:00.859 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.858+0000 7f73ffd8d640 10 monclient: init 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 10 monclient(hunting): picked mon.noname-a con 0x7f73f80a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 10 monclient(hunting): picked mon.noname-b con 0x7f73f813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 10 monclient(hunting): picked mon.noname-c con 0x7f73f81430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f8142520 con 0x7f73f80a98f0 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f80b28a0 con 0x7f73f81430c0 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f80b2a30 con 0x7f73f813f710 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:00.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.859+0000 7f73ffd8d640 10 monclient(hunting): authenticate will time out at 6687.957132s 2024-03-25T19:26:00.867 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.868+0000 7f73fe32a640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f73f81430c0 0x7f73f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:51584/0 (socket says 172.21.15.60:51584) 2024-03-25T19:26:00.867 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.868+0000 7f73fe32a640 1 -- 172.21.15.60:0/2099412550 learned_addr learned my addr 172.21.15.60:0/2099412550 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:00.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.882+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441324840 0 0) 0x7f73f8142520 con 0x7f73f80a98f0 2024-03-25T19:26:00.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.882+0000 7f73fcb27640 10 monclient(hunting): _negotiate 2024-03-25T19:26:00.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.882+0000 7f73fcb27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:00.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.882+0000 7f73fcb27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:00.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.882+0000 7f73fcb27640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:26:00.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.882+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e4003520 con 0x7f73f80a98f0 2024-03-25T19:26:00.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.890+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036109845 0 0) 0x7f73f80b28a0 con 0x7f73f81430c0 2024-03-25T19:26:00.890 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.890+0000 7f73fcb27640 10 monclient(hunting): _negotiate 2024-03-25T19:26:00.890 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.890+0000 7f73fcb27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:00.890 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.890+0000 7f73fcb27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:00.890 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.890+0000 7f73fcb27640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:26:00.890 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.890+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f8142520 con 0x7f73f81430c0 2024-03-25T19:26:00.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.895+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714208464 0 0) 0x7f73f80b2a30 con 0x7f73f813f710 2024-03-25T19:26:00.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.895+0000 7f73fcb27640 10 monclient(hunting): _negotiate 2024-03-25T19:26:00.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.895+0000 7f73fcb27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:00.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.895+0000 7f73fcb27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:00.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.895+0000 7f73fcb27640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:26:00.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.895+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f80b28a0 con 0x7f73f813f710 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1230263727 0 0) 0x7f73f8142520 con 0x7f73f81430c0 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 >> v1:172.21.15.60:6790/0 conn(0x7f73f813f710 legacy=0x7f73f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 >> v1:172.21.15.60:6789/0 conn(0x7f73f80a98f0 legacy=0x7f73f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 10 monclient: _finish_hunting 0 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 1 monclient: found mon.noname-c 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f80b44c0 con 0x7f73f81430c0 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 10 monclient: _finish_auth 0 2024-03-25T19:26:00.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 10 monclient: _check_auth_tickets 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f73f00019f0 con 0x7f73f81430c0 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73fcb27640 10 monclient: dump: 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:00.903 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:00.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73ffd8d640 5 monclient: authenticate success, global_id 4471 2024-03-25T19:26:00.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.902+0000 7f73ffd8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:00.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73f0001b50 con 0x7f73f81430c0 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73fcb27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73fcb27640 1 -- 172.21.15.60:0/2099412550 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f73f0001f60 con 0x7f73f81430c0 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73fcb27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73fcb27640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73fcb27640 10 monclient: dump: 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:00.920 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:00.921 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:00.921 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:00.921 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:00.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73ffd8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:00.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.920+0000 7f73ffd8d640 4 set_mon_vals no callback set 2024-03-25T19:26:00.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.921+0000 7f73ffd8d640 10 monclient: shutdown 2024-03-25T19:26:00.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.921+0000 7f73ffd8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:00.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.921+0000 7f73ffd8d640 1 -- 172.21.15.60:0/2099412550 >> v1:172.21.15.90:6789/0 conn(0x7f73f81430c0 legacy=0x7f73f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:00.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.921+0000 7f73ffd8d640 1 -- 172.21.15.60:0/2099412550 shutdown_connections 2024-03-25T19:26:00.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.921+0000 7f73ffd8d640 1 -- 172.21.15.60:0/2099412550 >> 172.21.15.60:0/2099412550 conn(0x7f73f80b7ea0 msgr2=0x7f73f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:00.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.921+0000 7f73ffd8d640 1 -- 172.21.15.60:0/2099412550 shutdown_connections 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.921+0000 7f73ffd8d640 1 -- 172.21.15.60:0/2099412550 wait complete. 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 10 monclient: build_initial_monmap 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 10 monclient: monmap: 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 1 librados: starting msgr at 2024-03-25T19:26:00.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 1 librados: starting objecter 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 1 Processor -- start 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 1 -- start start 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 1 librados: setting wanted keys 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 1 librados: calling monclient init 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.922+0000 7f73ffd8d640 10 monclient: init 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 10 monclient(hunting): picked mon.noname-b con 0x7f73f80a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 10 monclient(hunting): picked mon.noname-a con 0x7f73f813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 10 monclient(hunting): picked mon.noname-c con 0x7f73f81430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f8136560 con 0x7f73f813f710 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f80ab8b0 con 0x7f73f80a98f0 2024-03-25T19:26:00.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f81db660 con 0x7f73f81430c0 2024-03-25T19:26:00.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:00.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.923+0000 7f73ffd8d640 10 monclient(hunting): authenticate will time out at 6688.021502s 2024-03-25T19:26:00.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.925+0000 7f73fe32a640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f73f81430c0 0x7f73f81375b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:48534/0 (socket says 172.21.15.60:48534) 2024-03-25T19:26:00.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.925+0000 7f73fe32a640 1 -- 172.21.15.60:0/1614554787 learned_addr learned my addr 172.21.15.60:0/1614554787 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:00.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.929+0000 7f73ee7fc640 10 client.?.objecter ms_handle_connect 0x7f73f813f710 2024-03-25T19:26:00.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.929+0000 7f73ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:00.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.930+0000 7f73ee7fc640 10 client.?.objecter ms_handle_connect 0x7f73f81430c0 2024-03-25T19:26:00.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.930+0000 7f73ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:00.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.933+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008288833 0 0) 0x7f73f80b28a0 con 0x7f73f813f710 2024-03-25T19:26:00.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.933+0000 7f73ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:00.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.933+0000 7f73ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:00.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.933+0000 7f73ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:00.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.933+0000 7f73ee7fc640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:26:00.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.933+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73dc004eb0 con 0x7f73f813f710 2024-03-25T19:26:00.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.939+0000 7f73ee7fc640 10 client.?.objecter ms_handle_connect 0x7f73f80a98f0 2024-03-25T19:26:00.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.939+0000 7f73ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:00.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.945+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380484071 0 0) 0x7f73f81db660 con 0x7f73f81430c0 2024-03-25T19:26:00.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.945+0000 7f73ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:00.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.945+0000 7f73ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:00.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.945+0000 7f73ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:00.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.945+0000 7f73ee7fc640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:26:00.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.945+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f80b28a0 con 0x7f73f81430c0 2024-03-25T19:26:00.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.953+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3911804148 0 0) 0x7f73dc004eb0 con 0x7f73f813f710 2024-03-25T19:26:00.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.953+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f81db660 con 0x7f73f813f710 2024-03-25T19:26:00.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.953+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f73e8001750 con 0x7f73f813f710 2024-03-25T19:26:00.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.953+0000 7f73ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:00.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.953+0000 7f73ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:00.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.953+0000 7f73ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.953+0000 7f73ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.954+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3424740512 0 0) 0x7f73f80b28a0 con 0x7f73f81430c0 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.954+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73dc004eb0 con 0x7f73f81430c0 2024-03-25T19:26:00.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.954+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f73f0001ea0 con 0x7f73f81430c0 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.954+0000 7f73ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.954+0000 7f73ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.954+0000 7f73ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:00.955 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:00.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.958+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952822542 0 0) 0x7f73f80ab8b0 con 0x7f73f80a98f0 2024-03-25T19:26:00.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.958+0000 7f73ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:00.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.958+0000 7f73ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:00.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.958+0000 7f73ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:00.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.958+0000 7f73ee7fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:26:00.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.958+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f80b28a0 con 0x7f73f80a98f0 2024-03-25T19:26:00.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.967+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3483344561 0 0) 0x7f73f80b28a0 con 0x7f73f80a98f0 2024-03-25T19:26:00.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.967+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f80ab8b0 con 0x7f73f80a98f0 2024-03-25T19:26:00.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.969+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f73f4001910 con 0x7f73f80a98f0 2024-03-25T19:26:00.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.969+0000 7f73ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:00.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.969+0000 7f73ee7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:00.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.969+0000 7f73ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:26:00.970 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2007261805 0 0) 0x7f73f81db660 con 0x7f73f813f710 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 >> v1:172.21.15.60:6790/0 conn(0x7f73f81430c0 legacy=0x7f73f81375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 >> v1:172.21.15.90:6789/0 conn(0x7f73f80a98f0 legacy=0x7f73f81dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:26:00.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 1 monclient: found mon.a 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f81db380 con 0x7f73f813f710 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 5 monclient: authenticate success, global_id 4485 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 10 monclient: _renew_subs 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f80b0d50 con 0x7f73f813f710 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 10 monclient: _renew_subs 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f80b0750 con 0x7f73f813f710 2024-03-25T19:26:00.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.970+0000 7f73ffd8d640 1 librados: init done 2024-03-25T19:26:00.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.971+0000 7f73ffd8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:00.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.971+0000 7f73ffd8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:00.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.971+0000 7f73ffd8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:00.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:00.971+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f80627b0 con 0x7f73f813f710 2024-03-25T19:26:01.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.293+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73e8001c60 con 0x7f73f813f710 2024-03-25T19:26:01.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.294+0000 7f73ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:01.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.294+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f73e80037b0 con 0x7f73f813f710 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.294+0000 7f73ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.294+0000 7f73ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.294+0000 7f73ee7fc640 10 monclient: dump: 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:01.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.294+0000 7f73effff640 4 set_mon_vals no callback set 2024-03-25T19:26:01.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.294+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f73e8019780 con 0x7f73f813f710 2024-03-25T19:26:01.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.295+0000 7f73fdb29640 10 monclient: get_auth_request con 0x7f73dc06e090 auth_method 0 2024-03-25T19:26:01.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.295+0000 7f73ee7fc640 10 client.4485.objecter ms_handle_connect 0x7f73dc06e090 2024-03-25T19:26:01.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f73e8053470 con 0x7f73f813f710 2024-03-25T19:26:01.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 10 client.4485.objecter ms_dispatch 0x7f73f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:01.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:01.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:01.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-25T19:26:01.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73e804edb0 con 0x7f73f813f710 2024-03-25T19:26:01.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:01.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.300+0000 7f73ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:01.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.565+0000 7f73ffd8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:01.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.565+0000 7f73ffd8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:01.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.565+0000 7f73ffd8d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:01.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.565+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73f8141a00 con 0x7f73f813f710 2024-03-25T19:26:01.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.596+0000 7f73ee7fc640 1 -- 172.21.15.60:0/1614554787 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f73e8050eb0 con 0x7f73f813f710 2024-03-25T19:26:01.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.596+0000 7f73ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:01.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.596+0000 7f73ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:01.597 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:01.597 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 10 librados: watch_flush enter 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 10 librados: watch_flush exit 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 10 client.4485.objecter successfully canceled tick 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 >> v1:172.21.15.90:6800/1827638754 conn(0x7f73dc06e090 legacy=0x7f73dc070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 10 monclient: shutdown 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:01.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 >> v1:172.21.15.60:6789/0 conn(0x7f73f813f710 legacy=0x7f73f80b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:01.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 shutdown_connections 2024-03-25T19:26:01.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 >> 172.21.15.60:0/1614554787 conn(0x7f73f80b7ea0 msgr2=0x7f73f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:01.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 shutdown_connections 2024-03-25T19:26:01.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 1 -- 172.21.15.60:0/1614554787 wait complete. 2024-03-25T19:26:01.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:01.599+0000 7f73ffd8d640 1 librados: shutdown 2024-03-25T19:26:01.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:08.417 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.534+0000 7f714475d640 10 monclient: get_monmap_and_config 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.534+0000 7f714475d640 10 monclient: build_initial_monmap 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.534+0000 7f714475d640 10 monclient: monmap: 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:08.535445+0000 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:08.535445+0000 2024-03-25T19:26:08.534 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:08.535 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:08.535 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:08.535 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:08.535 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:08.535 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.535+0000 7f714475d640 1 Processor -- start 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.535+0000 7f714475d640 1 -- start start 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.535+0000 7f714475d640 10 monclient: init 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 10 monclient(hunting): picked mon.noname-c con 0x7f713c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 10 monclient(hunting): picked mon.noname-a con 0x7f713c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 10 monclient(hunting): picked mon.noname-b con 0x7f713c14be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0af940 con 0x7f713c0b1730 2024-03-25T19:26:08.536 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0b0a70 con 0x7f713c0b4d60 2024-03-25T19:26:08.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0b0d50 con 0x7f713c14be90 2024-03-25T19:26:08.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:08.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.536+0000 7f714475d640 10 monclient(hunting): authenticate will time out at 6695.634481s 2024-03-25T19:26:08.540 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.541+0000 7f71424f9640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f713c0b4d60 0x7f713c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:33644/0 (socket says 172.21.15.60:33644) 2024-03-25T19:26:08.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.541+0000 7f71424f9640 1 -- 172.21.15.60:0/3250537890 learned_addr learned my addr 172.21.15.60:0/3250537890 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:08.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.552+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309081631 0 0) 0x7f713c0b0d50 con 0x7f713c14be90 2024-03-25T19:26:08.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.552+0000 7f71414f7640 10 monclient(hunting): _negotiate 2024-03-25T19:26:08.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.552+0000 7f71414f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:08.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.552+0000 7f71414f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:08.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.552+0000 7f71414f7640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:26:08.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.552+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7128003520 con 0x7f713c14be90 2024-03-25T19:26:08.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.565+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090227743 0 0) 0x7f713c0b0a70 con 0x7f713c0b4d60 2024-03-25T19:26:08.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.565+0000 7f71414f7640 10 monclient(hunting): _negotiate 2024-03-25T19:26:08.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.565+0000 7f71414f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:08.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.565+0000 7f71414f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:08.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.565+0000 7f71414f7640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:26:08.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.565+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c0b0d50 con 0x7f713c0b4d60 2024-03-25T19:26:08.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 188354006 0 0) 0x7f713c0b0d50 con 0x7f713c0b4d60 2024-03-25T19:26:08.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 >> v1:172.21.15.60:6790/0 conn(0x7f713c14be90 legacy=0x7f713c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:08.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 >> v1:172.21.15.60:6789/0 conn(0x7f713c0b1730 legacy=0x7f713c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:26:08.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 10 monclient: _finish_hunting 0 2024-03-25T19:26:08.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 1 monclient: found mon.noname-c 2024-03-25T19:26:08.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0b1030 con 0x7f713c0b4d60 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 10 monclient: _finish_auth 0 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 10 monclient: _check_auth_tickets 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f71380019f0 con 0x7f713c0b4d60 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f714475d640 5 monclient: authenticate success, global_id 4477 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f714475d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f71414f7640 10 monclient: dump: 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:08.577 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:08.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.576+0000 7f714475d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:08.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.591+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7138001b50 con 0x7f713c0b4d60 2024-03-25T19:26:08.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f71414f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f71414f7640 1 -- 172.21.15.60:0/3250537890 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7138001fa0 con 0x7f713c0b4d60 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f71414f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f71414f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f71414f7640 10 monclient: dump: 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:08.592 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:08.593 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:08.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f714475d640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:08.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f714475d640 4 set_mon_vals no callback set 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f714475d640 10 monclient: shutdown 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f714475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.592+0000 7f714475d640 1 -- 172.21.15.60:0/3250537890 >> v1:172.21.15.90:6789/0 conn(0x7f713c0b4d60 legacy=0x7f713c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.593+0000 7f714475d640 1 -- 172.21.15.60:0/3250537890 shutdown_connections 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.593+0000 7f714475d640 1 -- 172.21.15.60:0/3250537890 >> 172.21.15.60:0/3250537890 conn(0x7f713c0617a0 msgr2=0x7f713c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.593+0000 7f714475d640 1 -- 172.21.15.60:0/3250537890 shutdown_connections 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.593+0000 7f714475d640 1 -- 172.21.15.60:0/3250537890 wait complete. 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 10 monclient: build_initial_monmap 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 10 monclient: monmap: 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:08.595 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 1 librados: starting msgr at 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 1 librados: starting objecter 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 1 Processor -- start 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 1 -- start start 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 1 librados: setting wanted keys 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 1 librados: calling monclient init 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.594+0000 7f714475d640 10 monclient: init 2024-03-25T19:26:08.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 10 monclient(hunting): picked mon.noname-c con 0x7f713c0b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 10 monclient(hunting): picked mon.noname-b con 0x7f713c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 10 monclient(hunting): picked mon.noname-a con 0x7f713c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c1422a0 con 0x7f713c14be90 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c142430 con 0x7f713c0b4d60 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c142660 con 0x7f713c0b1730 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.595+0000 7f714475d640 10 monclient(hunting): authenticate will time out at 6695.693250s 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.598+0000 7f7141cf8640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f713c0b4d60 0x7f713c1d7520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:33678/0 (socket says 172.21.15.60:33678) 2024-03-25T19:26:08.597 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.598+0000 7f7141cf8640 1 -- 172.21.15.60:0/1413456958 learned_addr learned my addr 172.21.15.60:0/1413456958 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:08.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.599+0000 7f7132ffd640 10 client.?.objecter ms_handle_connect 0x7f713c0b4d60 2024-03-25T19:26:08.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.599+0000 7f7132ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:08.604 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.605+0000 7f7132ffd640 10 client.?.objecter ms_handle_connect 0x7f713c0b1730 2024-03-25T19:26:08.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.605+0000 7f7132ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:08.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.676+0000 7f7132ffd640 10 client.?.objecter ms_handle_connect 0x7f713c14be90 2024-03-25T19:26:08.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.676+0000 7f7132ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:08.687 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.688+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160813121 0 0) 0x7f713c1422a0 con 0x7f713c14be90 2024-03-25T19:26:08.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.688+0000 7f7132ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:08.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.688+0000 7f7132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:08.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.688+0000 7f7132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:08.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.688+0000 7f7132ffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:26:08.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.688+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7120004eb0 con 0x7f713c14be90 2024-03-25T19:26:08.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.716+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2916801094 0 0) 0x7f7120004eb0 con 0x7f713c14be90 2024-03-25T19:26:08.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.716+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f713c1422a0 con 0x7f713c14be90 2024-03-25T19:26:08.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.716+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7134001890 con 0x7f713c14be90 2024-03-25T19:26:08.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.716+0000 7f7132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:08.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.716+0000 7f7132ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:08.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.716+0000 7f7132ffd640 10 monclient(hunting): dump: 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:08.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.716+0000 7f7132ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:08.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.720+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300278022 0 0) 0x7f713c142430 con 0x7f713c0b4d60 2024-03-25T19:26:08.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.720+0000 7f7132ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:08.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.720+0000 7f7132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:08.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.720+0000 7f7132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:08.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.720+0000 7f7132ffd640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:26:08.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.720+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7120004eb0 con 0x7f713c0b4d60 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 951663355 0 0) 0x7f713c1422a0 con 0x7f713c14be90 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 >> v1:172.21.15.60:6790/0 conn(0x7f713c0b1730 legacy=0x7f713c137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 >> v1:172.21.15.90:6789/0 conn(0x7f713c0b4d60 legacy=0x7f713c1d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 1 monclient: found mon.a 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c1427f0 con 0x7f713c14be90 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 10 monclient: _finish_auth 0 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f7132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:08.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 5 monclient: authenticate success, global_id 4488 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 10 monclient: _renew_subs 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c0a9470 con 0x7f713c14be90 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 10 monclient: _renew_subs 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c138860 con 0x7f713c14be90 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.730+0000 7f714475d640 1 librados: init done 2024-03-25T19:26:08.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.731+0000 7f714475d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:08.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.731+0000 7f714475d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:08.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.731+0000 7f714475d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:08.732 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:08.731+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c0aa890 con 0x7f713c14be90 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.058+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7134001d60 con 0x7f713c14be90 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.058+0000 7f7132ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.058+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f71340037b0 con 0x7f713c14be90 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.058+0000 7f7132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.058+0000 7f7132ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.058+0000 7f7132ffd640 10 monclient: dump: 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:09.058 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:09.059 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.058+0000 7f7140cf6640 4 set_mon_vals no callback set 2024-03-25T19:26:09.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.059+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f71340039d0 con 0x7f713c14be90 2024-03-25T19:26:09.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.060+0000 7f71424f9640 10 monclient: get_auth_request con 0x7f712006dd40 auth_method 0 2024-03-25T19:26:09.060 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.060+0000 7f7132ffd640 10 client.4488.objecter ms_handle_connect 0x7f712006dd40 2024-03-25T19:26:09.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f7134053480 con 0x7f713c14be90 2024-03-25T19:26:09.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 10 client.4488.objecter ms_dispatch 0x7f713c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:09.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:09.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:09.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-25T19:26:09.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f713404edc0 con 0x7f713c14be90 2024-03-25T19:26:09.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:09.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.065+0000 7f7132ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:09.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.328+0000 7f714475d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:09.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.328+0000 7f714475d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:09.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.328+0000 7f714475d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:09.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.328+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f713c0627b0 con 0x7f713c14be90 2024-03-25T19:26:09.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.379+0000 7f7132ffd640 1 -- 172.21.15.60:0/1413456958 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f7134050ec0 con 0x7f713c14be90 2024-03-25T19:26:09.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.379+0000 7f7132ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:09.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.379+0000 7f7132ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:09.379 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:09.379 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:09.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 10 librados: watch_flush enter 2024-03-25T19:26:09.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 10 librados: watch_flush exit 2024-03-25T19:26:09.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-25T19:26:09.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 10 client.4488.objecter successfully canceled tick 2024-03-25T19:26:09.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 >> v1:172.21.15.90:6800/1827638754 conn(0x7f712006dd40 legacy=0x7f7120070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:09.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 10 monclient: shutdown 2024-03-25T19:26:09.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:09.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 >> v1:172.21.15.60:6789/0 conn(0x7f713c14be90 legacy=0x7f713c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:09.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 shutdown_connections 2024-03-25T19:26:09.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 >> 172.21.15.60:0/1413456958 conn(0x7f713c0617a0 msgr2=0x7f713c0b66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:09.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 shutdown_connections 2024-03-25T19:26:09.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 1 -- 172.21.15.60:0/1413456958 wait complete. 2024-03-25T19:26:09.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:09.381+0000 7f714475d640 1 librados: shutdown 2024-03-25T19:26:09.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:16.199 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:16.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.313+0000 7fb20ff38640 10 monclient: get_monmap_and_config 2024-03-25T19:26:16.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.313+0000 7fb20ff38640 10 monclient: build_initial_monmap 2024-03-25T19:26:16.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.313+0000 7fb20ff38640 10 monclient: monmap: 2024-03-25T19:26:16.314 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:16.314 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:16.314 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:16.315353+0000 2024-03-25T19:26:16.314 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:16.315353+0000 2024-03-25T19:26:16.315 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:16.315 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:16.315 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:16.315 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:16.315 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:16.315 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:16.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.314+0000 7fb20ff38640 1 Processor -- start 2024-03-25T19:26:16.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.314+0000 7fb20ff38640 1 -- start start 2024-03-25T19:26:16.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.314+0000 7fb20ff38640 10 monclient: init 2024-03-25T19:26:16.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:16.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:16.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 10 monclient(hunting): picked mon.noname-a con 0x7fb2080a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:16.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 10 monclient(hunting): picked mon.noname-b con 0x7fb20813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:16.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 10 monclient(hunting): picked mon.noname-c con 0x7fb2081430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:16.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb208142520 con 0x7fb2080a98f0 2024-03-25T19:26:16.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080b28a0 con 0x7fb2081430c0 2024-03-25T19:26:16.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080b2a30 con 0x7fb20813f710 2024-03-25T19:26:16.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:16.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.315+0000 7fb20ff38640 10 monclient(hunting): authenticate will time out at 6703.414450s 2024-03-25T19:26:16.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.323+0000 7fb20e4d5640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fb2081430c0 0x7fb20814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:35648/0 (socket says 172.21.15.60:35648) 2024-03-25T19:26:16.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.323+0000 7fb20e4d5640 1 -- 172.21.15.60:0/47499462 learned_addr learned my addr 172.21.15.60:0/47499462 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720623608 0 0) 0x7fb2080b2a30 con 0x7fb20813f710 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): _negotiate 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f4003520 con 0x7fb20813f710 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337017915 0 0) 0x7fb2080b28a0 con 0x7fb2081430c0 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): _negotiate 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:26:16.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.333+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2080b2a30 con 0x7fb2081430c0 2024-03-25T19:26:16.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1096408631 0 0) 0x7fb1f4003520 con 0x7fb20813f710 2024-03-25T19:26:16.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 >> v1:172.21.15.90:6789/0 conn(0x7fb2081430c0 legacy=0x7fb20814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 >> v1:172.21.15.60:6789/0 conn(0x7fb2080a98f0 legacy=0x7fb2080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 10 monclient: _finish_hunting 0 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 1 monclient: found mon.noname-b 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080b44c0 con 0x7fb20813f710 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 10 monclient: _finish_auth 0 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 10 monclient: _check_auth_tickets 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb1f8001690 con 0x7fb20813f710 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ff38640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ff38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:16.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.347+0000 7fb20ccd2640 10 monclient: dump: 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:16.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.348+0000 7fb20ff38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:16.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1f80017f0 con 0x7fb20813f710 2024-03-25T19:26:16.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ccd2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:16.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ccd2640 1 -- 172.21.15.60:0/47499462 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb1f8001c00 con 0x7fb20813f710 2024-03-25T19:26:16.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ccd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:16.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ccd2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ccd2640 10 monclient: dump: 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ff38640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:16.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ff38640 4 set_mon_vals no callback set 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ff38640 10 monclient: shutdown 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ff38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ff38640 1 -- 172.21.15.60:0/47499462 >> v1:172.21.15.60:6790/0 conn(0x7fb20813f710 legacy=0x7fb2081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ff38640 1 -- 172.21.15.60:0/47499462 shutdown_connections 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.372+0000 7fb20ff38640 1 -- 172.21.15.60:0/47499462 >> 172.21.15.60:0/47499462 conn(0x7fb2080b7ea0 msgr2=0x7fb2080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.373+0000 7fb20ff38640 1 -- 172.21.15.60:0/47499462 shutdown_connections 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.373+0000 7fb20ff38640 1 -- 172.21.15.60:0/47499462 wait complete. 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.373+0000 7fb20ff38640 10 monclient: build_initial_monmap 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.373+0000 7fb20ff38640 10 monclient: monmap: 2024-03-25T19:26:16.375 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.374+0000 7fb20ff38640 1 librados: starting msgr at 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.374+0000 7fb20ff38640 1 librados: starting objecter 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.374+0000 7fb20ff38640 1 Processor -- start 2024-03-25T19:26:16.376 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.374+0000 7fb20ff38640 1 -- start start 2024-03-25T19:26:16.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.374+0000 7fb20ff38640 1 librados: setting wanted keys 2024-03-25T19:26:16.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.374+0000 7fb20ff38640 1 librados: calling monclient init 2024-03-25T19:26:16.377 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.374+0000 7fb20ff38640 10 monclient: init 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 10 monclient(hunting): picked mon.noname-a con 0x7fb2080a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 10 monclient(hunting): picked mon.noname-c con 0x7fb20813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 10 monclient(hunting): picked mon.noname-b con 0x7fb2081430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb208138730 con 0x7fb2080a98f0 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb208138a00 con 0x7fb2081430c0 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2080b1790 con 0x7fb20813f710 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:16.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.375+0000 7fb20ff38640 10 monclient(hunting): authenticate will time out at 6703.474420s 2024-03-25T19:26:16.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.379+0000 7fb20d4d3640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fb20813f710 0x7fb208134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:46540/0 (socket says 172.21.15.60:46540) 2024-03-25T19:26:16.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.379+0000 7fb20d4d3640 1 -- 172.21.15.60:0/1048300238 learned_addr learned my addr 172.21.15.60:0/1048300238 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:16.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.384+0000 7fb1fe7fc640 10 client.?.objecter ms_handle_connect 0x7fb2081430c0 2024-03-25T19:26:16.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.384+0000 7fb1fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:16.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.384+0000 7fb1fe7fc640 10 client.?.objecter ms_handle_connect 0x7fb20813f710 2024-03-25T19:26:16.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.384+0000 7fb1fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:16.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.394+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502912180 0 0) 0x7fb2080b1790 con 0x7fb20813f710 2024-03-25T19:26:16.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.394+0000 7fb1fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:16.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.394+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:16.394 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.394+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:16.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.394+0000 7fb1fe7fc640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:26:16.395 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.394+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1ec004eb0 con 0x7fb20813f710 2024-03-25T19:26:16.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.400+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454682009 0 0) 0x7fb208138a00 con 0x7fb2081430c0 2024-03-25T19:26:16.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.400+0000 7fb1fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:16.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.400+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:16.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.400+0000 7fb1fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:16.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.400+0000 7fb1fe7fc640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:26:16.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.400+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2080b1790 con 0x7fb2081430c0 2024-03-25T19:26:16.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.401+0000 7fb1fe7fc640 10 client.?.objecter ms_handle_connect 0x7fb2080a98f0 2024-03-25T19:26:16.402 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.401+0000 7fb1fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:16.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.409+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531331846 0 0) 0x7fb2080b1790 con 0x7fb2081430c0 2024-03-25T19:26:16.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.409+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb208138a00 con 0x7fb2081430c0 2024-03-25T19:26:16.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.414+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb200001bf0 con 0x7fb2081430c0 2024-03-25T19:26:16.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.414+0000 7fb1fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:16.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.414+0000 7fb1fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:16.414 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.414+0000 7fb1fe7fc640 10 monclient(hunting): dump: 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:16.415 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.414+0000 7fb1fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3097019076 0 0) 0x7fb208138a00 con 0x7fb2081430c0 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 >> v1:172.21.15.60:6790/0 conn(0x7fb20813f710 legacy=0x7fb208134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 >> v1:172.21.15.60:6789/0 conn(0x7fb2080a98f0 legacy=0x7fb2081379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 1 monclient: found mon.b 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2080b12e0 con 0x7fb2081430c0 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb1fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:26:16.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 10 monclient: _renew_subs 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2080b1530 con 0x7fb2081430c0 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 10 monclient: _renew_subs 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb208134900 con 0x7fb2081430c0 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.418+0000 7fb20ff38640 1 librados: init done 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.419+0000 7fb20ff38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.419+0000 7fb20ff38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.419+0000 7fb20ff38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:16.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.419+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2080627b0 con 0x7fb2081430c0 2024-03-25T19:26:16.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.673+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb200004b60 con 0x7fb2081430c0 2024-03-25T19:26:16.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.673+0000 7fb1fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:16.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.673+0000 7fb1fffff640 4 set_mon_vals no callback set 2024-03-25T19:26:16.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.684+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fb200002100 con 0x7fb2081430c0 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.684+0000 7fb1fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.684+0000 7fb1fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.684+0000 7fb1fe7fc640 10 monclient: dump: 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:16.685 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:16.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.685+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fb20001ad20 con 0x7fb2081430c0 2024-03-25T19:26:16.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.686+0000 7fb20dcd4640 10 monclient: get_auth_request con 0x7fb1ec06dd20 auth_method 0 2024-03-25T19:26:16.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.686+0000 7fb1fe7fc640 10 client.4486.objecter ms_handle_connect 0x7fb1ec06dd20 2024-03-25T19:26:16.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.689+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fb200054b50 con 0x7fb2081430c0 2024-03-25T19:26:16.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.689+0000 7fb1fe7fc640 10 client.4486.objecter ms_dispatch 0x7fb208000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:16.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.689+0000 7fb1fe7fc640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:16.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.689+0000 7fb1fe7fc640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:16.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.689+0000 7fb1fe7fc640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T19:26:16.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.690+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb200050490 con 0x7fb2081430c0 2024-03-25T19:26:16.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.690+0000 7fb1fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:16.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.690+0000 7fb1fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:16.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.920+0000 7fb20ff38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:16.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.920+0000 7fb20ff38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:16.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.920+0000 7fb20ff38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:16.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.920+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb20813fdc0 con 0x7fb2081430c0 2024-03-25T19:26:16.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.947+0000 7fb1fe7fc640 1 -- 172.21.15.60:0/1048300238 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fb200052590 con 0x7fb2081430c0 2024-03-25T19:26:16.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.947+0000 7fb1fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:16.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.947+0000 7fb1fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:16.948 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:16.948 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.949+0000 7fb20ff38640 10 librados: watch_flush enter 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 10 librados: watch_flush exit 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 10 client.4486.objecter successfully canceled tick 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 >> v1:172.21.15.90:6800/1827638754 conn(0x7fb1ec06dd20 legacy=0x7fb1ec0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 10 monclient: shutdown 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 >> v1:172.21.15.90:6789/0 conn(0x7fb2081430c0 legacy=0x7fb2081da980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 shutdown_connections 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 >> 172.21.15.60:0/1048300238 conn(0x7fb2080b7ea0 msgr2=0x7fb208074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 shutdown_connections 2024-03-25T19:26:16.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 1 -- 172.21.15.60:0/1048300238 wait complete. 2024-03-25T19:26:16.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:16.950+0000 7fb20ff38640 1 librados: shutdown 2024-03-25T19:26:16.963 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:19.862 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:20.574 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.869 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:20.580 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.896 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:20.607 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.922 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:20.576 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.933 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:20.587 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.934 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:20.646 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.948 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:20.659 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.959 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:20.613 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:19.982 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:20.693 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:23.769 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:23.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.886+0000 7f23515f4640 10 monclient: get_monmap_and_config 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.886+0000 7f23515f4640 10 monclient: build_initial_monmap 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.886+0000 7f23515f4640 10 monclient: monmap: 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:23.888075+0000 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:23.888075+0000 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:23.887 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.886+0000 7f23515f4640 1 Processor -- start 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 1 -- start start 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient: init 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient(hunting): picked mon.noname-c con 0x7f234c0a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient(hunting): picked mon.noname-b con 0x7f234c13f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient(hunting): picked mon.noname-a con 0x7f234c1430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c142520 con 0x7f234c1430c0 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c0b28a0 con 0x7f234c0a98f0 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c0b2a30 con 0x7f234c13f710 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:23.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.887+0000 7f23515f4640 10 monclient(hunting): authenticate will time out at 6710.986627s 2024-03-25T19:26:23.898 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.898+0000 7f234a7fc640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f234c13f710 0x7f234c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:51296/0 (socket says 172.21.15.60:51296) 2024-03-25T19:26:23.898 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.898+0000 7f234a7fc640 1 -- 172.21.15.60:0/3201991129 learned_addr learned my addr 172.21.15.60:0/3201991129 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:23.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.906+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783018435 0 0) 0x7f234c0b2a30 con 0x7f234c13f710 2024-03-25T19:26:23.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.906+0000 7f2349ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:23.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.906+0000 7f2349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:23.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.906+0000 7f2349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:23.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.906+0000 7f2349ffb640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:26:23.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.906+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2334003520 con 0x7f234c13f710 2024-03-25T19:26:23.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.912+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517045843 0 0) 0x7f234c0b28a0 con 0x7f234c0a98f0 2024-03-25T19:26:23.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.912+0000 7f2349ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:23.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.912+0000 7f2349ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:23.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.912+0000 7f2349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:23.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.912+0000 7f2349ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:26:23.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.912+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c0b2a30 con 0x7f234c0a98f0 2024-03-25T19:26:23.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872424677 0 0) 0x7f2334003520 con 0x7f234c13f710 2024-03-25T19:26:23.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 >> v1:172.21.15.90:6789/0 conn(0x7f234c0a98f0 legacy=0x7f234c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 >> v1:172.21.15.60:6789/0 conn(0x7f234c1430c0 legacy=0x7f234c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 1 monclient: found mon.noname-b 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c0b44c0 con 0x7f234c13f710 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 10 monclient: _finish_auth 0 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2340001690 con 0x7f234c13f710 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.913+0000 7f2349ffb640 10 monclient: dump: 2024-03-25T19:26:23.915 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.914+0000 7f23515f4640 5 monclient: authenticate success, global_id 4484 2024-03-25T19:26:23.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.914+0000 7f23515f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23400017f0 con 0x7f234c13f710 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f2349ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f2349ffb640 1 -- 172.21.15.60:0/3201991129 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2340001c00 con 0x7f234c13f710 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f23515f4640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f23515f4640 4 set_mon_vals no callback set 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f2349ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f2349ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.921+0000 7f2349ffb640 10 monclient: dump: 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:23.922 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.923 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.923 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:23.923 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:23.923 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:23.923 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:23.923 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:23.923 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.922+0000 7f23515f4640 10 monclient: shutdown 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.922+0000 7f23515f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.922+0000 7f23515f4640 1 -- 172.21.15.60:0/3201991129 >> v1:172.21.15.60:6790/0 conn(0x7f234c13f710 legacy=0x7f234c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.922+0000 7f23515f4640 1 -- 172.21.15.60:0/3201991129 shutdown_connections 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.922+0000 7f23515f4640 1 -- 172.21.15.60:0/3201991129 >> 172.21.15.60:0/3201991129 conn(0x7f234c0b7ea0 msgr2=0x7f234c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.922+0000 7f23515f4640 1 -- 172.21.15.60:0/3201991129 shutdown_connections 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.922+0000 7f23515f4640 1 -- 172.21.15.60:0/3201991129 wait complete. 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 10 monclient: build_initial_monmap 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 10 monclient: monmap: 2024-03-25T19:26:23.925 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 1 librados: starting msgr at 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 1 librados: starting objecter 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 1 Processor -- start 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 1 -- start start 2024-03-25T19:26:23.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 1 librados: setting wanted keys 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 1 librados: calling monclient init 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.923+0000 7f23515f4640 10 monclient: init 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 10 monclient(hunting): picked mon.noname-a con 0x7f234c0a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 10 monclient(hunting): picked mon.noname-c con 0x7f234c13f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 10 monclient(hunting): picked mon.noname-b con 0x7f234c1430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c1378c0 con 0x7f234c0a98f0 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c138460 con 0x7f234c1430c0 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c138d30 con 0x7f234c13f710 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:23.927 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.924+0000 7f23515f4640 10 monclient(hunting): authenticate will time out at 6711.023259s 2024-03-25T19:26:23.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.934+0000 7f234a7fc640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f234c13f710 0x7f234c1db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:51302/0 (socket says 172.21.15.60:51302) 2024-03-25T19:26:23.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.934+0000 7f234a7fc640 1 -- 172.21.15.60:0/772242367 learned_addr learned my addr 172.21.15.60:0/772242367 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:23.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.935+0000 7f23337fe640 10 client.?.objecter ms_handle_connect 0x7f234c13f710 2024-03-25T19:26:23.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.935+0000 7f23337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:23.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.942+0000 7f23337fe640 10 client.?.objecter ms_handle_connect 0x7f234c1430c0 2024-03-25T19:26:23.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.942+0000 7f23337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.945+0000 7f23337fe640 10 client.?.objecter ms_handle_connect 0x7f234c0a98f0 2024-03-25T19:26:23.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.945+0000 7f23337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:23.956 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.956+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243335276 0 0) 0x7f234c138d30 con 0x7f234c13f710 2024-03-25T19:26:23.956 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.956+0000 7f23337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:23.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.956+0000 7f23337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:23.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.956+0000 7f23337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:23.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.956+0000 7f23337fe640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:26:23.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.956+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2338004eb0 con 0x7f234c13f710 2024-03-25T19:26:23.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.963+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154536686 0 0) 0x7f234c138460 con 0x7f234c1430c0 2024-03-25T19:26:23.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.963+0000 7f23337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:23.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.963+0000 7f23337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:23.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.963+0000 7f23337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:23.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.963+0000 7f23337fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T19:26:23.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.963+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c138d30 con 0x7f234c1430c0 2024-03-25T19:26:23.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.969+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691343332 0 0) 0x7f234c1378c0 con 0x7f234c0a98f0 2024-03-25T19:26:23.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.969+0000 7f23337fe640 10 monclient(hunting): _negotiate 2024-03-25T19:26:23.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.969+0000 7f23337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:23.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.969+0000 7f23337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:23.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.969+0000 7f23337fe640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:26:23.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.969+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234c138460 con 0x7f234c0a98f0 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.986+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418761001 0 0) 0x7f234c138d30 con 0x7f234c1430c0 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.986+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f234c1378c0 con 0x7f234c1430c0 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.986+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2344001b10 con 0x7f234c1430c0 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.986+0000 7f23337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.986+0000 7f23337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.986+0000 7f23337fe640 10 monclient(hunting): dump: 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.987 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.988 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:23.988 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:23.988 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:23.988 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:23.988 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:23.988 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:23.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.986+0000 7f23337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:23.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.994+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232657027 0 0) 0x7f2338004eb0 con 0x7f234c13f710 2024-03-25T19:26:23.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.994+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f234c138d30 con 0x7f234c13f710 2024-03-25T19:26:23.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.994+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2340001bc0 con 0x7f234c13f710 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.994+0000 7f23337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.994+0000 7f23337fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:23.994+0000 7f23337fe640 10 monclient(hunting): dump: 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:23.995 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:24.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2584358585 0 0) 0x7f234c1378c0 con 0x7f234c1430c0 2024-03-25T19:26:24.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 >> v1:172.21.15.60:6790/0 conn(0x7f234c13f710 legacy=0x7f234c1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:24.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 >> v1:172.21.15.60:6789/0 conn(0x7f234c0a98f0 legacy=0x7f234c137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 10 monclient: _finish_hunting 0 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 1 monclient: found mon.b 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c1dc070 con 0x7f234c1430c0 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 10 monclient: _finish_auth 0 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 10 monclient: _check_auth_tickets 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 5 monclient: authenticate success, global_id 4492 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 10 monclient: _renew_subs 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c138fa0 con 0x7f234c1430c0 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:24.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 10 monclient: _renew_subs 2024-03-25T19:26:24.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:24.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.001+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f234c139560 con 0x7f234c1430c0 2024-03-25T19:26:24.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.002+0000 7f23515f4640 1 librados: init done 2024-03-25T19:26:24.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.003+0000 7f23515f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:24.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.003+0000 7f23515f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:24.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.003+0000 7f23515f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:24.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.003+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f234c0627b0 con 0x7f234c1430c0 2024-03-25T19:26:24.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.309+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2344002000 con 0x7f234c1430c0 2024-03-25T19:26:24.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.309+0000 7f23337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.309+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2344003b30 con 0x7f234c1430c0 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.309+0000 7f23337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.309+0000 7f23337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.309+0000 7f23337fe640 10 monclient: dump: 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:24.310 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:24.311 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:24.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.309+0000 7f23497fa640 4 set_mon_vals no callback set 2024-03-25T19:26:24.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.310+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2344003d50 con 0x7f234c1430c0 2024-03-25T19:26:24.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.310+0000 7f234affd640 10 monclient: get_auth_request con 0x7f233806df70 auth_method 0 2024-03-25T19:26:24.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.311+0000 7f23337fe640 10 client.4492.objecter ms_handle_connect 0x7f233806df70 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f23440537c0 con 0x7f234c1430c0 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 10 client.4492.objecter ms_dispatch 0x7f234c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f234404f2d0 con 0x7f234c1430c0 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:24.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.315+0000 7f23337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:24.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.554+0000 7f23515f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:24.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.554+0000 7f23515f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:24.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.554+0000 7f23515f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:26:24.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.554+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f234c13bd30 con 0x7f234c1430c0 2024-03-25T19:26:24.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.648+0000 7f23337fe640 1 -- 172.21.15.60:0/772242367 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f2344051200 con 0x7f234c1430c0 2024-03-25T19:26:24.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.648+0000 7f23337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:24.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.648+0000 7f23337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:24.649 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:24.649 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:24.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 10 librados: watch_flush enter 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 10 librados: watch_flush exit 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 10 client.4492.objecter successfully canceled tick 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 >> v1:172.21.15.90:6800/1827638754 conn(0x7f233806df70 legacy=0x7f2338070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 10 monclient: shutdown 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 >> v1:172.21.15.90:6789/0 conn(0x7f234c1430c0 legacy=0x7f234c1db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 shutdown_connections 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 >> 172.21.15.60:0/772242367 conn(0x7f234c0b7ea0 msgr2=0x7f234c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 shutdown_connections 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 1 -- 172.21.15.60:0/772242367 wait complete. 2024-03-25T19:26:24.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:24.650+0000 7f23515f4640 1 librados: shutdown 2024-03-25T19:26:24.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:31.469 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:31.583 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.582+0000 7fedfd377640 10 monclient: get_monmap_and_config 2024-03-25T19:26:31.583 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.582+0000 7fedfd377640 10 monclient: build_initial_monmap 2024-03-25T19:26:31.583 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.582+0000 7fedfd377640 10 monclient: monmap: 2024-03-25T19:26:31.583 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:31.583 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:31.583 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:31.584450+0000 2024-03-25T19:26:31.583 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:31.584450+0000 2024-03-25T19:26:31.584 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:31.584 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:31.584 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:31.584 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:31.584 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:31.584 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.583+0000 7fedfd377640 1 Processor -- start 2024-03-25T19:26:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.583+0000 7fedfd377640 1 -- start start 2024-03-25T19:26:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.583+0000 7fedfd377640 10 monclient: init 2024-03-25T19:26:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:26:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 10 monclient(hunting): picked mon.noname-b con 0x7fedf80a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 10 monclient(hunting): picked mon.noname-a con 0x7fedf813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:31.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 10 monclient(hunting): picked mon.noname-c con 0x7fedf81430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:31.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf8142520 con 0x7fedf813f710 2024-03-25T19:26:31.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80b28a0 con 0x7fedf81430c0 2024-03-25T19:26:31.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80b2a30 con 0x7fedf80a98f0 2024-03-25T19:26:31.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:31.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.584+0000 7fedfd377640 10 monclient(hunting): authenticate will time out at 6718.683095s 2024-03-25T19:26:31.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.590+0000 7fedf77fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fedf81430c0 0x7fedf814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:46140/0 (socket says 172.21.15.60:46140) 2024-03-25T19:26:31.591 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.590+0000 7fedf77fe640 1 -- 172.21.15.60:0/1484175284 learned_addr learned my addr 172.21.15.60:0/1484175284 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:31.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.605+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497054590 0 0) 0x7fedf80b2a30 con 0x7fedf80a98f0 2024-03-25T19:26:31.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.605+0000 7fedf67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:31.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.605+0000 7fedf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:31.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.605+0000 7fedf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:31.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.605+0000 7fedf67fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:26:31.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.605+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede4003520 con 0x7fedf80a98f0 2024-03-25T19:26:31.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.612+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806224421 0 0) 0x7fedf80b28a0 con 0x7fedf81430c0 2024-03-25T19:26:31.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.612+0000 7fedf67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:26:31.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.612+0000 7fedf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:31.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.612+0000 7fedf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:31.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.612+0000 7fedf67fc640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:26:31.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.612+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf80b2a30 con 0x7fedf81430c0 2024-03-25T19:26:31.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.616+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887923513 0 0) 0x7fede4003520 con 0x7fedf80a98f0 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 >> v1:172.21.15.90:6789/0 conn(0x7fedf81430c0 legacy=0x7fedf814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 >> v1:172.21.15.60:6789/0 conn(0x7fedf813f710 legacy=0x7fedf81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 1 monclient: found mon.noname-b 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80b44c0 con 0x7fedf80a98f0 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 10 monclient: _finish_auth 0 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedfd377640 5 monclient: authenticate success, global_id 4490 2024-03-25T19:26:31.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fede8001690 con 0x7fedf80a98f0 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedfd377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedf67fc640 10 monclient: dump: 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:31.619 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:31.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.617+0000 7fedfd377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:31.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fede80017f0 con 0x7fedf80a98f0 2024-03-25T19:26:31.631 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedf67fc640 1 -- 172.21.15.60:0/1484175284 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fede8001c00 con 0x7fedf80a98f0 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedfd377640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedfd377640 4 set_mon_vals no callback set 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedf67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.631+0000 7fedf67fc640 10 monclient: dump: 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:31.632 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:31.633 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:31.633 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:31.633 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:31.633 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:31.633 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:31.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.632+0000 7fedfd377640 10 monclient: shutdown 2024-03-25T19:26:31.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.632+0000 7fedfd377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.632+0000 7fedfd377640 1 -- 172.21.15.60:0/1484175284 >> v1:172.21.15.60:6790/0 conn(0x7fedf80a98f0 legacy=0x7fedf80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.632+0000 7fedfd377640 1 -- 172.21.15.60:0/1484175284 shutdown_connections 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.632+0000 7fedfd377640 1 -- 172.21.15.60:0/1484175284 >> 172.21.15.60:0/1484175284 conn(0x7fedf80b7ea0 msgr2=0x7fedf80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.632+0000 7fedfd377640 1 -- 172.21.15.60:0/1484175284 shutdown_connections 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.632+0000 7fedfd377640 1 -- 172.21.15.60:0/1484175284 wait complete. 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.633+0000 7fedfd377640 10 monclient: build_initial_monmap 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.633+0000 7fedfd377640 10 monclient: monmap: 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:31.635 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.633+0000 7fedfd377640 1 librados: starting msgr at 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.633+0000 7fedfd377640 1 librados: starting objecter 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.633+0000 7fedfd377640 1 Processor -- start 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.634+0000 7fedfd377640 1 -- start start 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.634+0000 7fedfd377640 1 librados: setting wanted keys 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.634+0000 7fedfd377640 1 librados: calling monclient init 2024-03-25T19:26:31.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.634+0000 7fedfd377640 10 monclient: init 2024-03-25T19:26:31.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.634+0000 7fedfd377640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 10 monclient(hunting): picked mon.noname-b con 0x7fedf80a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 10 monclient(hunting): picked mon.noname-a con 0x7fedf813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 10 monclient(hunting): picked mon.noname-c con 0x7fedf81430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80ab9e0 con 0x7fedf813f710 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf8138c20 con 0x7fedf80a98f0 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf8138f00 con 0x7fedf81430c0 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.635+0000 7fedfd377640 10 monclient(hunting): authenticate will time out at 6718.733989s 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.636+0000 7fedee7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fedf813f710 0x7fedf81d7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:50104/0 (socket says 172.21.15.60:50104) 2024-03-25T19:26:31.638 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.637+0000 7fedee7fc640 1 -- 172.21.15.60:0/2324115121 learned_addr learned my addr 172.21.15.60:0/2324115121 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:31.639 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.639+0000 7fedeffff640 10 client.?.objecter ms_handle_connect 0x7fedf813f710 2024-03-25T19:26:31.639 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.639+0000 7fedeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:31.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.646+0000 7fedeffff640 10 client.?.objecter ms_handle_connect 0x7fedf80a98f0 2024-03-25T19:26:31.647 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.646+0000 7fedeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:31.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.650+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435248984 0 0) 0x7fedf80ab9e0 con 0x7fedf813f710 2024-03-25T19:26:31.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.650+0000 7fedeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:31.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.650+0000 7fedeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:31.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.650+0000 7fedeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:31.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.650+0000 7fedeffff640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:26:31.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.650+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd8004eb0 con 0x7fedf813f710 2024-03-25T19:26:31.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.652+0000 7fedeffff640 10 client.?.objecter ms_handle_connect 0x7fedf81430c0 2024-03-25T19:26:31.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.652+0000 7fedeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:31.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.658+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1580715112 0 0) 0x7fedf8138c20 con 0x7fedf80a98f0 2024-03-25T19:26:31.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.658+0000 7fedeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:31.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.658+0000 7fedeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:31.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.658+0000 7fedeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:31.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.658+0000 7fedeffff640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:26:31.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.658+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf80ab9e0 con 0x7fedf80a98f0 2024-03-25T19:26:31.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.662+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3936926690 0 0) 0x7fedd8004eb0 con 0x7fedf813f710 2024-03-25T19:26:31.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.662+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedf8138c20 con 0x7fedf813f710 2024-03-25T19:26:31.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.662+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fede0001a30 con 0x7fedf813f710 2024-03-25T19:26:31.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.662+0000 7fedeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:31.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.662+0000 7fedeffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:31.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.662+0000 7fedeffff640 10 monclient(hunting): dump: 2024-03-25T19:26:31.663 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:31.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.662+0000 7fedeffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:31.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.676+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597258964 0 0) 0x7fedf8138f00 con 0x7fedf81430c0 2024-03-25T19:26:31.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.676+0000 7fedeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:31.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.676+0000 7fedeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:31.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.676+0000 7fedeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:31.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.676+0000 7fedeffff640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:26:31.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.676+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedd8004eb0 con 0x7fedf81430c0 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.679+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3284313864 0 0) 0x7fedf8138c20 con 0x7fedf813f710 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.679+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 >> v1:172.21.15.60:6790/0 conn(0x7fedf81430c0 legacy=0x7fedf81e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.679+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 >> v1:172.21.15.90:6789/0 conn(0x7fedf80a98f0 legacy=0x7fedf8137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.679+0000 7fedeffff640 10 monclient: _finish_hunting 0 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.679+0000 7fedeffff640 1 monclient: found mon.a 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.679+0000 7fedeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedeffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf814e7a0 con 0x7fedf813f710 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedeffff640 10 monclient: _finish_auth 0 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedeffff640 10 monclient: _check_auth_tickets 2024-03-25T19:26:31.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 5 monclient: authenticate success, global_id 4506 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 10 monclient: _renew_subs 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf81386d0 con 0x7fedf813f710 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 10 monclient: _renew_subs 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf81d8060 con 0x7fedf813f710 2024-03-25T19:26:31.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.680+0000 7fedfd377640 1 librados: init done 2024-03-25T19:26:31.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.681+0000 7fedfd377640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:31.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.681+0000 7fedfd377640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:31.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.681+0000 7fedfd377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:31.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.681+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf80627b0 con 0x7fedf813f710 2024-03-25T19:26:31.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.899+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fede0001c70 con 0x7fedf813f710 2024-03-25T19:26:31.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.899+0000 7fedeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:31.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:31.899+0000 7fedf5ffb640 4 set_mon_vals no callback set 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.006+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fede0003a60 con 0x7fedf813f710 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.006+0000 7fedeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.006+0000 7fedeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.006+0000 7fedeffff640 10 monclient: dump: 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:32.007 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:32.008 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:32.008 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:32.008 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:32.008 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:32.008 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.007+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fede0003c40 con 0x7fedf813f710 2024-03-25T19:26:32.008 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.008+0000 7fedf6ffd640 10 monclient: get_auth_request con 0x7fedd806de00 auth_method 0 2024-03-25T19:26:32.008 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.008+0000 7fedeffff640 10 client.4506.objecter ms_handle_connect 0x7fedd806de00 2024-03-25T19:26:32.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fede0054910 con 0x7fedf813f710 2024-03-25T19:26:32.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 10 client.4506.objecter ms_dispatch 0x7fedf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:32.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:32.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:32.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-25T19:26:32.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fede0050250 con 0x7fedf813f710 2024-03-25T19:26:32.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:32.015 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.014+0000 7fedeffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:32.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.269+0000 7fedfd377640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:32.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.269+0000 7fedfd377640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:32.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.269+0000 7fedfd377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:32.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.269+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedf814f240 con 0x7fedf813f710 2024-03-25T19:26:32.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.304+0000 7fedeffff640 1 -- 172.21.15.60:0/2324115121 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fede0052350 con 0x7fedf813f710 2024-03-25T19:26:32.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.304+0000 7fedeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:32.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.304+0000 7fedeffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:32.305 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:32.306 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.306+0000 7fedfd377640 10 librados: watch_flush enter 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 10 librados: watch_flush exit 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 10 client.4506.objecter successfully canceled tick 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 >> v1:172.21.15.90:6800/1827638754 conn(0x7fedd806de00 legacy=0x7fedd80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 10 monclient: shutdown 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 >> v1:172.21.15.60:6789/0 conn(0x7fedf813f710 legacy=0x7fedf81d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 shutdown_connections 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 >> 172.21.15.60:0/2324115121 conn(0x7fedf80b7ea0 msgr2=0x7fedf8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:32.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 shutdown_connections 2024-03-25T19:26:32.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 1 -- 172.21.15.60:0/2324115121 wait complete. 2024-03-25T19:26:32.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:32.307+0000 7fedfd377640 1 librados: shutdown 2024-03-25T19:26:32.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:39.125 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:39.238 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.238+0000 7fd663a0f640 10 monclient: get_monmap_and_config 2024-03-25T19:26:39.238 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.238+0000 7fd663a0f640 10 monclient: build_initial_monmap 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.238+0000 7fd663a0f640 10 monclient: monmap: 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:39.240124+0000 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:39.240124+0000 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:39.239 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:39.240 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.239+0000 7fd663a0f640 1 Processor -- start 2024-03-25T19:26:39.240 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.239+0000 7fd663a0f640 1 -- start start 2024-03-25T19:26:39.240 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.239+0000 7fd663a0f640 10 monclient: init 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fd65c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fd65c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fd65c14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c0af940 con 0x7fd65c0b1730 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c0b0a70 con 0x7fd65c14be90 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c0b0d50 con 0x7fd65c0b4d60 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:39.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.240+0000 7fd663a0f640 10 monclient(hunting): authenticate will time out at 6726.339039s 2024-03-25T19:26:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.247+0000 7fd661fac640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd65c14be90 0x7fd65c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:60122/0 (socket says 172.21.15.60:60122) 2024-03-25T19:26:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.247+0000 7fd661fac640 1 -- 172.21.15.60:0/3185648839 learned_addr learned my addr 172.21.15.60:0/3185648839 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:39.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.271+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802097602 0 0) 0x7fd65c0b0a70 con 0x7fd65c14be90 2024-03-25T19:26:39.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.271+0000 7fd653fff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:39.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.271+0000 7fd653fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:39.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.271+0000 7fd653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:39.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.271+0000 7fd653fff640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:26:39.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.271+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd648003520 con 0x7fd65c14be90 2024-03-25T19:26:39.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.280+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034829003 0 0) 0x7fd65c0b0d50 con 0x7fd65c0b4d60 2024-03-25T19:26:39.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.280+0000 7fd653fff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:39.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.280+0000 7fd653fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:39.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.280+0000 7fd653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:39.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.280+0000 7fd653fff640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:26:39.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.280+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd65c0b0a70 con 0x7fd65c0b4d60 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063759896 0 0) 0x7fd648003520 con 0x7fd65c14be90 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 >> v1:172.21.15.60:6790/0 conn(0x7fd65c0b4d60 legacy=0x7fd65c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 >> v1:172.21.15.60:6789/0 conn(0x7fd65c0b1730 legacy=0x7fd65c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 10 monclient: _finish_hunting 0 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 1 monclient: found mon.noname-c 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c0b1030 con 0x7fd65c14be90 2024-03-25T19:26:39.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 10 monclient: _finish_auth 0 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 10 monclient: _check_auth_tickets 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd663a0f640 5 monclient: authenticate success, global_id 4501 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd663a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd654001a10 con 0x7fd65c14be90 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd653fff640 10 monclient: dump: 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:39.283 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:39.284 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:39.284 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:39.284 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:39.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.281+0000 7fd663a0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:39.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd654001b70 con 0x7fd65c14be90 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd653fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd653fff640 1 -- 172.21.15.60:0/3185648839 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd654001f80 con 0x7fd65c14be90 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd653fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd653fff640 10 monclient: dump: 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:39.303 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:39.304 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd663a0f640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.302+0000 7fd663a0f640 4 set_mon_vals no callback set 2024-03-25T19:26:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.303+0000 7fd663a0f640 10 monclient: shutdown 2024-03-25T19:26:39.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.303+0000 7fd663a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.303+0000 7fd663a0f640 1 -- 172.21.15.60:0/3185648839 >> v1:172.21.15.90:6789/0 conn(0x7fd65c14be90 legacy=0x7fd65c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.303+0000 7fd663a0f640 1 -- 172.21.15.60:0/3185648839 shutdown_connections 2024-03-25T19:26:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.303+0000 7fd663a0f640 1 -- 172.21.15.60:0/3185648839 >> 172.21.15.60:0/3185648839 conn(0x7fd65c0617a0 msgr2=0x7fd65c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.303+0000 7fd663a0f640 1 -- 172.21.15.60:0/3185648839 shutdown_connections 2024-03-25T19:26:39.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.303+0000 7fd663a0f640 1 -- 172.21.15.60:0/3185648839 wait complete. 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.304+0000 7fd663a0f640 10 monclient: build_initial_monmap 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.304+0000 7fd663a0f640 10 monclient: monmap: 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:39.306 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.305+0000 7fd663a0f640 1 librados: starting msgr at 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.305+0000 7fd663a0f640 1 librados: starting objecter 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.305+0000 7fd663a0f640 1 Processor -- start 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.305+0000 7fd663a0f640 1 -- start start 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.305+0000 7fd663a0f640 1 librados: setting wanted keys 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.305+0000 7fd663a0f640 1 librados: calling monclient init 2024-03-25T19:26:39.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.305+0000 7fd663a0f640 10 monclient: init 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fd65c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fd65c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fd65c14be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c13c5a0 con 0x7fd65c0b1730 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c1421b0 con 0x7fd65c0b4d60 2024-03-25T19:26:39.308 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65c140840 con 0x7fd65c14be90 2024-03-25T19:26:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:39.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.306+0000 7fd663a0f640 10 monclient(hunting): authenticate will time out at 6726.405523s 2024-03-25T19:26:39.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.312+0000 7fd6617ab640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fd65c0b1730 0x7fd65c13be90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53366/0 (socket says 172.21.15.60:53366) 2024-03-25T19:26:39.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.312+0000 7fd6617ab640 1 -- 172.21.15.60:0/699144802 learned_addr learned my addr 172.21.15.60:0/699144802 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.312+0000 7fd651ffb640 10 client.?.objecter ms_handle_connect 0x7fd65c0b1730 2024-03-25T19:26:39.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.312+0000 7fd651ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:39.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.322+0000 7fd651ffb640 10 client.?.objecter ms_handle_connect 0x7fd65c0b4d60 2024-03-25T19:26:39.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.322+0000 7fd651ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:39.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.323+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703917731 0 0) 0x7fd65c13c5a0 con 0x7fd65c0b1730 2024-03-25T19:26:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.323+0000 7fd651ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.323+0000 7fd651ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.323+0000 7fd651ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.323+0000 7fd651ffb640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:26:39.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.323+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd640004eb0 con 0x7fd65c0b1730 2024-03-25T19:26:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.331+0000 7fd651ffb640 10 client.?.objecter ms_handle_connect 0x7fd65c14be90 2024-03-25T19:26:39.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.331+0000 7fd651ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.332+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 684425234 0 0) 0x7fd640004eb0 con 0x7fd65c0b1730 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.332+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd65c13c5a0 con 0x7fd65c0b1730 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.332+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd658001a10 con 0x7fd65c0b1730 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.332+0000 7fd651ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.332+0000 7fd651ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.332+0000 7fd651ffb640 10 monclient(hunting): dump: 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:39.333 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:39.334 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:39.334 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:39.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.332+0000 7fd651ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:39.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3111866258 0 0) 0x7fd65c13c5a0 con 0x7fd65c0b1730 2024-03-25T19:26:39.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 >> v1:172.21.15.60:6790/0 conn(0x7fd65c14be90 legacy=0x7fd65c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:39.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 >> v1:172.21.15.90:6789/0 conn(0x7fd65c0b4d60 legacy=0x7fd65c1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:39.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:26:39.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 1 monclient: found mon.a 2024-03-25T19:26:39.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:39.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c1d7da0 con 0x7fd65c0b1730 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 10 monclient: _finish_auth 0 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd651ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 10 monclient: _renew_subs 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65c141240 con 0x7fd65c0b1730 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 10 monclient: _renew_subs 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65c140560 con 0x7fd65c0b1730 2024-03-25T19:26:39.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.344+0000 7fd663a0f640 1 librados: init done 2024-03-25T19:26:39.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.345+0000 7fd663a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:39.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.345+0000 7fd663a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:39.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.345+0000 7fd663a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:39.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.345+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd65c0afbd0 con 0x7fd65c0b1730 2024-03-25T19:26:39.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.655+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd658001c90 con 0x7fd65c0b1730 2024-03-25T19:26:39.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.656+0000 7fd651ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:39.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.656+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd658003a60 con 0x7fd65c0b1730 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.656+0000 7fd651ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.656+0000 7fd651ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.656+0000 7fd651ffb640 10 monclient: dump: 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:39.657 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:39.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.656+0000 7fd6537fe640 4 set_mon_vals no callback set 2024-03-25T19:26:39.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.656+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fd658003c40 con 0x7fd65c0b1730 2024-03-25T19:26:39.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.657+0000 7fd660faa640 10 monclient: get_auth_request con 0x7fd64006dd70 auth_method 0 2024-03-25T19:26:39.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.657+0000 7fd651ffb640 10 client.4512.objecter ms_handle_connect 0x7fd64006dd70 2024-03-25T19:26:39.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.662+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fd658054860 con 0x7fd65c0b1730 2024-03-25T19:26:39.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.662+0000 7fd651ffb640 10 client.4512.objecter ms_dispatch 0x7fd65c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:39.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.662+0000 7fd651ffb640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:39.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.662+0000 7fd651ffb640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:39.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.663+0000 7fd651ffb640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T19:26:39.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.663+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd658050370 con 0x7fd65c0b1730 2024-03-25T19:26:39.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.663+0000 7fd651ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:39.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.663+0000 7fd651ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:39.878 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:40.589 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:39.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.899+0000 7fd663a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:39.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.899+0000 7fd663a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:39.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.899+0000 7fd663a0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:39.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.899+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd65c1409d0 con 0x7fd65c0b1730 2024-03-25T19:26:39.926 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:40.580 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:39.929 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:40.583 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:39.930 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:40.641 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:39.953 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:06:40.664 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:39.980 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:06:40.633 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:39.992 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:40.703 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:39.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.995+0000 7fd651ffb640 1 -- 172.21.15.60:0/699144802 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fd6580522a0 con 0x7fd65c0b1730 2024-03-25T19:26:39.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.995+0000 7fd651ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:39.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.995+0000 7fd651ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:39.996 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:39.996 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.997+0000 7fd663a0f640 10 librados: watch_flush enter 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.997+0000 7fd663a0f640 10 librados: watch_flush exit 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.997+0000 7fd663a0f640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.997+0000 7fd663a0f640 10 client.4512.objecter successfully canceled tick 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 >> v1:172.21.15.90:6800/1827638754 conn(0x7fd64006dd70 legacy=0x7fd640070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 10 monclient: shutdown 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 >> v1:172.21.15.60:6789/0 conn(0x7fd65c0b1730 legacy=0x7fd65c13be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 shutdown_connections 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 >> 172.21.15.60:0/699144802 conn(0x7fd65c0617a0 msgr2=0x7fd65c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:39.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 shutdown_connections 2024-03-25T19:26:40.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 1 -- 172.21.15.60:0/699144802 wait complete. 2024-03-25T19:26:40.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:39.998+0000 7fd663a0f640 1 librados: shutdown 2024-03-25T19:26:40.006 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:40.718 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:40.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:40.028 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:06:40.739 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:26:46.816 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.933+0000 7ff82264a640 10 monclient: get_monmap_and_config 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.933+0000 7ff82264a640 10 monclient: build_initial_monmap 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.933+0000 7ff82264a640 10 monclient: monmap: 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:46.935211+0000 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:46.935211+0000 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:46.934 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:46.935 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.934+0000 7ff82264a640 1 Processor -- start 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.934+0000 7ff82264a640 1 -- start start 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.934+0000 7ff82264a640 10 monclient: init 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 10 monclient(hunting): picked mon.noname-a con 0x7ff81c0a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 10 monclient(hunting): picked mon.noname-b con 0x7ff81c13f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 10 monclient(hunting): picked mon.noname-c con 0x7ff81c1430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81c142520 con 0x7ff81c0a98f0 2024-03-25T19:26:46.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81c0b28a0 con 0x7ff81c1430c0 2024-03-25T19:26:46.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81c0b2a30 con 0x7ff81c13f710 2024-03-25T19:26:46.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:46.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.935+0000 7ff82264a640 10 monclient(hunting): authenticate will time out at 6734.034403s 2024-03-25T19:26:46.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.937+0000 7ff81b7fe640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7ff81c13f710 0x7ff81c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:48186/0 (socket says 172.21.15.60:48186) 2024-03-25T19:26:46.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.937+0000 7ff81b7fe640 1 -- 172.21.15.60:0/1238319849 learned_addr learned my addr 172.21.15.60:0/1238319849 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:46.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.943+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020380586 0 0) 0x7ff81c0b2a30 con 0x7ff81c13f710 2024-03-25T19:26:46.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.943+0000 7ff81affd640 10 monclient(hunting): _negotiate 2024-03-25T19:26:46.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.943+0000 7ff81affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:46.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.943+0000 7ff81affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:46.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.943+0000 7ff81affd640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:26:46.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.943+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff804003520 con 0x7ff81c13f710 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.955+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 154874504 0 0) 0x7ff804003520 con 0x7ff81c13f710 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.955+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 >> v1:172.21.15.90:6789/0 conn(0x7ff81c1430c0 legacy=0x7ff81c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.955+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 >> v1:172.21.15.60:6789/0 conn(0x7ff81c0a98f0 legacy=0x7ff81c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.955+0000 7ff81affd640 10 monclient: _finish_hunting 0 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 1 monclient: found mon.noname-b 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81c0b44c0 con 0x7ff81c13f710 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 10 monclient: _finish_auth 0 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 10 monclient: _check_auth_tickets 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff810001690 con 0x7ff81c13f710 2024-03-25T19:26:46.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff82264a640 5 monclient: authenticate success, global_id 4502 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff82264a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff81affd640 10 monclient: dump: 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:46.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.956+0000 7ff82264a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff810001860 con 0x7ff81c13f710 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff81affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff81affd640 1 -- 172.21.15.60:0/1238319849 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff810001cb0 con 0x7ff81c13f710 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff81affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff81affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff81affd640 10 monclient: dump: 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:46.965 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:46.966 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:46.966 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:46.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff82264a640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:46.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.964+0000 7ff82264a640 4 set_mon_vals no callback set 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.965+0000 7ff82264a640 10 monclient: shutdown 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.965+0000 7ff82264a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.965+0000 7ff82264a640 1 -- 172.21.15.60:0/1238319849 >> v1:172.21.15.60:6790/0 conn(0x7ff81c13f710 legacy=0x7ff81c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.965+0000 7ff82264a640 1 -- 172.21.15.60:0/1238319849 shutdown_connections 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.965+0000 7ff82264a640 1 -- 172.21.15.60:0/1238319849 >> 172.21.15.60:0/1238319849 conn(0x7ff81c0b7ea0 msgr2=0x7ff81c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.965+0000 7ff82264a640 1 -- 172.21.15.60:0/1238319849 shutdown_connections 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.965+0000 7ff82264a640 1 -- 172.21.15.60:0/1238319849 wait complete. 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 10 monclient: build_initial_monmap 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 10 monclient: monmap: 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:46.968 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 1 librados: starting msgr at 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 1 librados: starting objecter 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 1 Processor -- start 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 1 -- start start 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 1 librados: setting wanted keys 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 1 librados: calling monclient init 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.966+0000 7ff82264a640 10 monclient: init 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:46.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 10 monclient(hunting): picked mon.noname-b con 0x7ff81c0a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 10 monclient(hunting): picked mon.noname-c con 0x7ff81c13f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 10 monclient(hunting): picked mon.noname-a con 0x7ff81c1430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81c138cc0 con 0x7ff81c1430c0 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81c132cf0 con 0x7ff81c0a98f0 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81c137310 con 0x7ff81c13f710 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:46.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.967+0000 7ff82264a640 10 monclient(hunting): authenticate will time out at 6734.066352s 2024-03-25T19:26:46.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.974+0000 7ff820be7640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff81c1430c0 0x7ff81c1325e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:37206/0 (socket says 172.21.15.60:37206) 2024-03-25T19:26:46.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.974+0000 7ff820be7640 1 -- 172.21.15.60:0/1944923086 learned_addr learned my addr 172.21.15.60:0/1944923086 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:46.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.975+0000 7ff818ff9640 10 client.?.objecter ms_handle_connect 0x7ff81c1430c0 2024-03-25T19:26:46.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.975+0000 7ff818ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:46.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.978+0000 7ff818ff9640 10 client.?.objecter ms_handle_connect 0x7ff81c13f710 2024-03-25T19:26:46.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.978+0000 7ff818ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:46.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209482227 0 0) 0x7ff81c137310 con 0x7ff81c13f710 2024-03-25T19:26:46.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:26:46.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:46.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:46.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:26:46.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff808004eb0 con 0x7ff81c13f710 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 10 client.?.objecter ms_handle_connect 0x7ff81c0a98f0 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.983+0000 7ff818ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.984+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256948140 0 0) 0x7ff81c138cc0 con 0x7ff81c1430c0 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.984+0000 7ff818ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.984+0000 7ff818ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.984+0000 7ff818ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.984+0000 7ff818ff9640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:26:46.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.984+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff81c137310 con 0x7ff81c1430c0 2024-03-25T19:26:46.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.989+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225747025 0 0) 0x7ff81c132cf0 con 0x7ff81c0a98f0 2024-03-25T19:26:46.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.989+0000 7ff818ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:26:46.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.989+0000 7ff818ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:46.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.989+0000 7ff818ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:46.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.989+0000 7ff818ff9640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:26:46.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.989+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff81c138cc0 con 0x7ff81c0a98f0 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.996+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2983098648 0 0) 0x7ff808004eb0 con 0x7ff81c13f710 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.996+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff81c132cf0 con 0x7ff81c13f710 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.996+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff810001b70 con 0x7ff81c13f710 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.996+0000 7ff818ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.996+0000 7ff818ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.996+0000 7ff818ff9640 10 monclient(hunting): dump: 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:46.997 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:46.998 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:46.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:46.996+0000 7ff818ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.000+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 572057604 0 0) 0x7ff81c137310 con 0x7ff81c1430c0 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.000+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff808004eb0 con 0x7ff81c1430c0 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.000+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff814001c10 con 0x7ff81c1430c0 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.000+0000 7ff818ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.000+0000 7ff818ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.000+0000 7ff818ff9640 10 monclient(hunting): dump: 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:47.001 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:47.002 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:47.002 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:47.002 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:47.002 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:47.002 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:47.002 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:47.002 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:47.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.009+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2890232224 0 0) 0x7ff81c138cc0 con 0x7ff81c0a98f0 2024-03-25T19:26:47.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.009+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff81c137310 con 0x7ff81c0a98f0 2024-03-25T19:26:47.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.009+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff80c0017d0 con 0x7ff81c0a98f0 2024-03-25T19:26:47.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.009+0000 7ff818ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.009+0000 7ff818ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.009+0000 7ff818ff9640 10 monclient(hunting): dump: 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:47.010 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:47.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.009+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3343527159 0 0) 0x7ff81c132cf0 con 0x7ff81c13f710 2024-03-25T19:26:47.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 >> v1:172.21.15.90:6789/0 conn(0x7ff81c0a98f0 legacy=0x7ff81c137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:47.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 >> v1:172.21.15.60:6789/0 conn(0x7ff81c1430c0 legacy=0x7ff81c1325e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:47.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:26:47.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 1 monclient: found mon.c 2024-03-25T19:26:47.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:47.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff81c138ef0 con 0x7ff81c13f710 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 10 monclient: _finish_auth 0 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff818ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 5 monclient: authenticate success, global_id 4505 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 10 monclient: _renew_subs 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81c1391d0 con 0x7ff81c13f710 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 10 monclient: _renew_subs 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c139560 con 0x7ff81c13f710 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.010+0000 7ff82264a640 1 librados: init done 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.011+0000 7ff82264a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:47.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.011+0000 7ff82264a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:47.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.011+0000 7ff82264a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:26:47.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.011+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81c138300 con 0x7ff81c13f710 2024-03-25T19:26:47.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.327+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff810001010 con 0x7ff81c13f710 2024-03-25T19:26:47.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.327+0000 7ff818ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:47.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.327+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff810001cf0 con 0x7ff81c13f710 2024-03-25T19:26:47.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.327+0000 7ff818ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:47.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.327+0000 7ff818ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:47.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.327+0000 7ff818ff9640 10 monclient: dump: 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:47.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.327+0000 7ff81a7fc640 4 set_mon_vals no callback set 2024-03-25T19:26:47.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.328+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7ff81001aa60 con 0x7ff81c13f710 2024-03-25T19:26:47.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.329+0000 7ff81bfff640 10 monclient: get_auth_request con 0x7ff80806e010 auth_method 0 2024-03-25T19:26:47.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.329+0000 7ff818ff9640 10 client.4505.objecter ms_handle_connect 0x7ff80806e010 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7ff810054720 con 0x7ff81c13f710 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 10 client.4505.objecter ms_dispatch 0x7ff81c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff810050060 con 0x7ff81c13f710 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:47.336 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.335+0000 7ff818ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:47.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.592+0000 7ff82264a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:47.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.592+0000 7ff82264a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:47.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.592+0000 7ff82264a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:26:47.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.592+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff81c139360 con 0x7ff81c13f710 2024-03-25T19:26:47.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.618+0000 7ff818ff9640 1 -- 172.21.15.60:0/1944923086 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7ff810052160 con 0x7ff81c13f710 2024-03-25T19:26:47.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.618+0000 7ff818ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:47.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.618+0000 7ff818ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:47.619 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:47.619 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:47.621 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.620+0000 7ff82264a640 10 librados: watch_flush enter 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 10 librados: watch_flush exit 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 10 client.4505.objecter successfully canceled tick 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 >> v1:172.21.15.90:6800/1827638754 conn(0x7ff80806e010 legacy=0x7ff8080704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 10 monclient: shutdown 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 >> v1:172.21.15.60:6790/0 conn(0x7ff81c13f710 legacy=0x7ff81c131ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 shutdown_connections 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 >> 172.21.15.60:0/1944923086 conn(0x7ff81c0b7ea0 msgr2=0x7ff81c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:47.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 shutdown_connections 2024-03-25T19:26:47.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 1 -- 172.21.15.60:0/1944923086 wait complete. 2024-03-25T19:26:47.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:47.621+0000 7ff82264a640 1 librados: shutdown 2024-03-25T19:26:47.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:26:54.440 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:26:54.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.553+0000 7fa68b6ba640 10 monclient: get_monmap_and_config 2024-03-25T19:26:54.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.553+0000 7fa68b6ba640 10 monclient: build_initial_monmap 2024-03-25T19:26:54.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.554+0000 7fa68b6ba640 10 monclient: monmap: 2024-03-25T19:26:54.553 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:54.553 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:54.553 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:26:54.554679+0000 2024-03-25T19:26:54.554 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:26:54.554679+0000 2024-03-25T19:26:54.554 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:54.554 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:54.554 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:54.554 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:26:54.554 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:26:54.554 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:54.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.554+0000 7fa68b6ba640 1 Processor -- start 2024-03-25T19:26:54.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.554+0000 7fa68b6ba640 1 -- start start 2024-03-25T19:26:54.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.554+0000 7fa68b6ba640 10 monclient: init 2024-03-25T19:26:54.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:54.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:54.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa6840b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:54.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa6840b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:54.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa68414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:54.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6840af940 con 0x7fa68414be90 2024-03-25T19:26:54.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6840b0a70 con 0x7fa6840b1730 2024-03-25T19:26:54.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6840b0d50 con 0x7fa6840b4d60 2024-03-25T19:26:54.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:54.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.555+0000 7fa68b6ba640 10 monclient(hunting): authenticate will time out at 6741.653772s 2024-03-25T19:26:54.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.563+0000 7fa689456640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fa6840b4d60 0x7fa68413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:53064/0 (socket says 172.21.15.60:53064) 2024-03-25T19:26:54.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.563+0000 7fa689456640 1 -- 172.21.15.60:0/1576765182 learned_addr learned my addr 172.21.15.60:0/1576765182 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:54.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.635+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796478111 0 0) 0x7fa6840af940 con 0x7fa68414be90 2024-03-25T19:26:54.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.635+0000 7fa67bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:26:54.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.635+0000 7fa67bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:54.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.635+0000 7fa67bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:54.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.635+0000 7fa67bfff640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:26:54.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.635+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa670003520 con 0x7fa68414be90 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262159199 0 0) 0x7fa670003520 con 0x7fa68414be90 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 >> v1:172.21.15.60:6790/0 conn(0x7fa6840b4d60 legacy=0x7fa68413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 >> v1:172.21.15.90:6789/0 conn(0x7fa6840b1730 legacy=0x7fa6840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 1 monclient: found mon.noname-a 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6840b1030 con 0x7fa68414be90 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 10 monclient: _finish_auth 0 2024-03-25T19:26:54.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa67c001a10 con 0x7fa68414be90 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa67bfff640 10 monclient: dump: 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:54.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa68b6ba640 5 monclient: authenticate success, global_id 4518 2024-03-25T19:26:54.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.675+0000 7fa68b6ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:26:54.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa67c001b70 con 0x7fa68414be90 2024-03-25T19:26:54.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa67bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:54.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa67bfff640 1 -- 172.21.15.60:0/1576765182 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa67c001f80 con 0x7fa68414be90 2024-03-25T19:26:54.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa67bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:54.702 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa67bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa67bfff640 10 monclient: dump: 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa68b6ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:26:54.703 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.703+0000 7fa68b6ba640 4 set_mon_vals no callback set 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.704+0000 7fa68b6ba640 10 monclient: shutdown 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.704+0000 7fa68b6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.704+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1576765182 >> v1:172.21.15.60:6789/0 conn(0x7fa68414be90 legacy=0x7fa68414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.704+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1576765182 shutdown_connections 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.704+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1576765182 >> 172.21.15.60:0/1576765182 conn(0x7fa6840617a0 msgr2=0x7fa684061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.704+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1576765182 shutdown_connections 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.704+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1576765182 wait complete. 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 10 monclient: build_initial_monmap 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 10 monclient: monmap: 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:26:54.705 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 1 librados: starting msgr at 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 1 librados: starting objecter 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 1 Processor -- start 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 1 -- start start 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 1 librados: setting wanted keys 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 1 librados: calling monclient init 2024-03-25T19:26:54.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.705+0000 7fa68b6ba640 10 monclient: init 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa6840b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa6840b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa68414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa684140380 con 0x7fa68414be90 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa684140660 con 0x7fa6840b4d60 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa684141e30 con 0x7fa6840b1730 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:26:54.708 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.706+0000 7fa68b6ba640 10 monclient(hunting): authenticate will time out at 6741.804721s 2024-03-25T19:26:54.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.710+0000 7fa688c55640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa6840b4d60 0x7fa684142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:43576/0 (socket says 172.21.15.60:43576) 2024-03-25T19:26:54.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.710+0000 7fa688c55640 1 -- 172.21.15.60:0/1798277625 learned_addr learned my addr 172.21.15.60:0/1798277625 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:26:54.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.715+0000 7fa679ffb640 10 client.?.objecter ms_handle_connect 0x7fa6840b4d60 2024-03-25T19:26:54.715 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.715+0000 7fa679ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:54.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.720+0000 7fa679ffb640 10 client.?.objecter ms_handle_connect 0x7fa68414be90 2024-03-25T19:26:54.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.720+0000 7fa679ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:54.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.720+0000 7fa679ffb640 10 client.?.objecter ms_handle_connect 0x7fa6840b1730 2024-03-25T19:26:54.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.720+0000 7fa679ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:26:54.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.726+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042414337 0 0) 0x7fa684141e30 con 0x7fa6840b1730 2024-03-25T19:26:54.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.726+0000 7fa679ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:54.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.726+0000 7fa679ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:54.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.726+0000 7fa679ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:54.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.726+0000 7fa679ffb640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:26:54.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.726+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa668004eb0 con 0x7fa6840b1730 2024-03-25T19:26:54.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.728+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234380905 0 0) 0x7fa684140380 con 0x7fa68414be90 2024-03-25T19:26:54.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.728+0000 7fa679ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:54.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.728+0000 7fa679ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:54.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.728+0000 7fa679ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:54.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.728+0000 7fa679ffb640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:26:54.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.728+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa684141e30 con 0x7fa68414be90 2024-03-25T19:26:54.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.731+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783712271 0 0) 0x7fa684140660 con 0x7fa6840b4d60 2024-03-25T19:26:54.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.731+0000 7fa679ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:26:54.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.731+0000 7fa679ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:26:54.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.731+0000 7fa679ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:26:54.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.731+0000 7fa679ffb640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:26:54.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.731+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa684140380 con 0x7fa6840b4d60 2024-03-25T19:26:54.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.741+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2333938291 0 0) 0x7fa684141e30 con 0x7fa68414be90 2024-03-25T19:26:54.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.742+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa684140660 con 0x7fa68414be90 2024-03-25T19:26:54.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.742+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa67c001ec0 con 0x7fa68414be90 2024-03-25T19:26:54.741 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.742+0000 7fa679ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.742+0000 7fa679ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.742+0000 7fa679ffb640 10 monclient(hunting): dump: 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:54.742 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.742+0000 7fa679ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:26:54.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.748+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693871740 0 0) 0x7fa684140380 con 0x7fa6840b4d60 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.748+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa684141e30 con 0x7fa6840b4d60 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.748+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa6740018f0 con 0x7fa6840b4d60 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.748+0000 7fa679ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.748+0000 7fa679ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.748+0000 7fa679ffb640 10 monclient(hunting): dump: 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:54.748 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:54.749 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:54.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.750+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284335369 0 0) 0x7fa668004eb0 con 0x7fa6840b1730 2024-03-25T19:26:54.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.750+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa684140380 con 0x7fa6840b1730 2024-03-25T19:26:54.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.750+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa6800018a0 con 0x7fa6840b1730 2024-03-25T19:26:54.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.750+0000 7fa679ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:54.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.750+0000 7fa679ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:26:54.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.750+0000 7fa679ffb640 10 monclient(hunting): dump: 2024-03-25T19:26:54.750 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:54.751 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:54.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 264746072 0 0) 0x7fa684140660 con 0x7fa68414be90 2024-03-25T19:26:54.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 >> v1:172.21.15.60:6790/0 conn(0x7fa6840b1730 legacy=0x7fa684138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:54.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 >> v1:172.21.15.90:6789/0 conn(0x7fa6840b4d60 legacy=0x7fa684142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:54.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:26:54.755 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 1 monclient: found mon.a 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684142110 con 0x7fa68414be90 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 10 monclient: _finish_auth 0 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa679ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 5 monclient: authenticate success, global_id 4521 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 10 monclient: _renew_subs 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa684140e60 con 0x7fa68414be90 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 10 monclient: _renew_subs 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa684142c40 con 0x7fa68414be90 2024-03-25T19:26:54.756 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.755+0000 7fa68b6ba640 1 librados: init done 2024-03-25T19:26:54.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.756+0000 7fa68b6ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:54.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.756+0000 7fa68b6ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:54.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.756+0000 7fa68b6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:54.757 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:54.756+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6840627b0 con 0x7fa68414be90 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.124+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa67c0020c0 con 0x7fa68414be90 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.124+0000 7fa679ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.124+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa67c003dc0 con 0x7fa68414be90 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.124+0000 7fa679ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.124+0000 7fa679ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.124+0000 7fa679ffb640 10 monclient: dump: 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:26:55.124 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.124+0000 7fa67b7fe640 4 set_mon_vals no callback set 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.125+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fa67c019d90 con 0x7fa68414be90 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.126+0000 7fa689456640 10 monclient: get_auth_request con 0x7fa66806e0f0 auth_method 0 2024-03-25T19:26:55.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.126+0000 7fa679ffb640 10 client.4521.objecter ms_handle_connect 0x7fa66806e0f0 2024-03-25T19:26:55.132 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fa67c053860 con 0x7fa68414be90 2024-03-25T19:26:55.132 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 10 client.4521.objecter ms_dispatch 0x7fa684000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:26:55.132 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:26:55.132 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:26:55.132 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T19:26:55.132 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa67c04f1a0 con 0x7fa68414be90 2024-03-25T19:26:55.133 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:26:55.133 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.132+0000 7fa679ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:26:55.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.384+0000 7fa68b6ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:55.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.384+0000 7fa68b6ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:55.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.384+0000 7fa68b6ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:26:55.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.384+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6840b5cf0 con 0x7fa68414be90 2024-03-25T19:26:55.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.413+0000 7fa679ffb640 1 -- 172.21.15.60:0/1798277625 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fa67c0512a0 con 0x7fa68414be90 2024-03-25T19:26:55.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.413+0000 7fa679ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:26:55.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.413+0000 7fa679ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:26:55.413 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:26:55.413 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.415+0000 7fa68b6ba640 10 librados: watch_flush enter 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 10 librados: watch_flush exit 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 10 client.4521.objecter successfully canceled tick 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 >> v1:172.21.15.90:6800/1827638754 conn(0x7fa66806e0f0 legacy=0x7fa6680705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 10 monclient: shutdown 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 >> v1:172.21.15.60:6789/0 conn(0x7fa68414be90 legacy=0x7fa6841e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:26:55.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 shutdown_connections 2024-03-25T19:26:55.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 >> 172.21.15.60:0/1798277625 conn(0x7fa6840617a0 msgr2=0x7fa6840b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:26:55.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 shutdown_connections 2024-03-25T19:26:55.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 1 -- 172.21.15.60:0/1798277625 wait complete. 2024-03-25T19:26:55.417 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:26:55.416+0000 7fa68b6ba640 1 librados: shutdown 2024-03-25T19:26:55.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:00.191 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:00.902 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.191 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:00.845 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.192 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:00.846 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.204 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:00.915 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.226 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:00.880 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.244 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:00.955 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.268 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:00.980 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.277 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:00.989 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:00.312 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:01.023 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:02.235 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:02.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.353+0000 7f8e236fb640 10 monclient: get_monmap_and_config 2024-03-25T19:27:02.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.353+0000 7f8e236fb640 10 monclient: build_initial_monmap 2024-03-25T19:27:02.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.353+0000 7f8e236fb640 10 monclient: monmap: 2024-03-25T19:27:02.353 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:02.353 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:02.353 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:02.354403+0000 2024-03-25T19:27:02.353 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:02.354403+0000 2024-03-25T19:27:02.354 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:02.354 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:02.354 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:02.354 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:02.354 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:02.354 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:02.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.354+0000 7f8e236fb640 1 Processor -- start 2024-03-25T19:27:02.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.354+0000 7f8e236fb640 1 -- start start 2024-03-25T19:27:02.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.354+0000 7f8e236fb640 10 monclient: init 2024-03-25T19:27:02.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c0a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c13f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c1430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c142520 con 0x7f8e1c13f710 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c0b28a0 con 0x7f8e1c1430c0 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c0b2a30 con 0x7f8e1c0a98f0 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:02.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.355+0000 7f8e236fb640 10 monclient(hunting): authenticate will time out at 6749.453558s 2024-03-25T19:27:02.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.369+0000 7f8e20c96640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f8e1c13f710 0x7f8e1c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38828/0 (socket says 172.21.15.60:38828) 2024-03-25T19:27:02.369 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.369+0000 7f8e20c96640 1 -- 172.21.15.60:0/719001471 learned_addr learned my addr 172.21.15.60:0/719001471 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273779049 0 0) 0x7f8e1c0b2a30 con 0x7f8e1c0a98f0 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e08003520 con 0x7f8e1c0a98f0 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432838013 0 0) 0x7f8e1c142520 con 0x7f8e1c13f710 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:02.382 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:02.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:02.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 10 monclient(hunting): my global_id is 4524 2024-03-25T19:27:02.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.382+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e1c0b2a30 con 0x7f8e1c13f710 2024-03-25T19:27:02.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.385+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253696418 0 0) 0x7f8e1c0b28a0 con 0x7f8e1c1430c0 2024-03-25T19:27:02.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.385+0000 7f8e13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:02.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.385+0000 7f8e13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:02.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.385+0000 7f8e13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:02.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.385+0000 7f8e13fff640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:27:02.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.385+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e1c142520 con 0x7f8e1c1430c0 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2033569764 0 0) 0x7f8e1c0b2a30 con 0x7f8e1c13f710 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 >> v1:172.21.15.60:6790/0 conn(0x7f8e1c0a98f0 legacy=0x7f8e1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 >> v1:172.21.15.90:6789/0 conn(0x7f8e1c1430c0 legacy=0x7f8e1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 1 monclient: found mon.noname-a 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c0b44c0 con 0x7f8e1c13f710 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:02.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8e0c001670 con 0x7f8e1c13f710 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e236fb640 5 monclient: authenticate success, global_id 4524 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e236fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e13fff640 10 monclient: dump: 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:02.399 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:02.400 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:02.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.398+0000 7f8e236fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e0c0017d0 con 0x7f8e1c13f710 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e13fff640 1 -- 172.21.15.60:0/719001471 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8e0c001be0 con 0x7f8e1c13f710 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e236fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e236fb640 4 set_mon_vals no callback set 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:02.416 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.416+0000 7f8e13fff640 10 monclient: dump: 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:02.417 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.417+0000 7f8e236fb640 10 monclient: shutdown 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.417+0000 7f8e236fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.417+0000 7f8e236fb640 1 -- 172.21.15.60:0/719001471 >> v1:172.21.15.60:6789/0 conn(0x7f8e1c13f710 legacy=0x7f8e1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.417+0000 7f8e236fb640 1 -- 172.21.15.60:0/719001471 shutdown_connections 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.417+0000 7f8e236fb640 1 -- 172.21.15.60:0/719001471 >> 172.21.15.60:0/719001471 conn(0x7f8e1c0b7ea0 msgr2=0x7f8e1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.417+0000 7f8e236fb640 1 -- 172.21.15.60:0/719001471 shutdown_connections 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.417+0000 7f8e236fb640 1 -- 172.21.15.60:0/719001471 wait complete. 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.418+0000 7f8e236fb640 10 monclient: build_initial_monmap 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.418+0000 7f8e236fb640 10 monclient: monmap: 2024-03-25T19:27:02.420 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.418+0000 7f8e236fb640 1 librados: starting msgr at 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.418+0000 7f8e236fb640 1 librados: starting objecter 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.418+0000 7f8e236fb640 1 Processor -- start 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 1 -- start start 2024-03-25T19:27:02.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 1 librados: setting wanted keys 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 1 librados: calling monclient init 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 10 monclient: init 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8e1c0a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 10 monclient(hunting): picked mon.noname-a con 0x7f8e1c13f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8e1c1430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c13c530 con 0x7f8e1c13f710 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.419+0000 7f8e236fb640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c12f860 con 0x7f8e1c0a98f0 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.420+0000 7f8e236fb640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c12fad0 con 0x7f8e1c1430c0 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.420+0000 7f8e236fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:02.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.420+0000 7f8e236fb640 10 monclient(hunting): authenticate will time out at 6749.517897s 2024-03-25T19:27:02.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.426+0000 7f8e21497640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f8e1c0a98f0 0x7f8e1c0b7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:48470/0 (socket says 172.21.15.60:48470) 2024-03-25T19:27:02.425 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.426+0000 7f8e21497640 1 -- 172.21.15.60:0/1760030376 learned_addr learned my addr 172.21.15.60:0/1760030376 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:02.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.427+0000 7f8e11ffb640 10 client.?.objecter ms_handle_connect 0x7f8e1c0a98f0 2024-03-25T19:27:02.426 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.427+0000 7f8e11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:02.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.438+0000 7f8e11ffb640 10 client.?.objecter ms_handle_connect 0x7f8e1c1430c0 2024-03-25T19:27:02.438 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.438+0000 7f8e11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:02.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.440+0000 7f8e11ffb640 10 client.?.objecter ms_handle_connect 0x7f8e1c13f710 2024-03-25T19:27:02.440 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.440+0000 7f8e11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:02.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.508+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112129457 0 0) 0x7f8e1c13c530 con 0x7f8e1c13f710 2024-03-25T19:27:02.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.508+0000 7f8e11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:02.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.508+0000 7f8e11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:02.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.508+0000 7f8e11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:02.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.508+0000 7f8e11ffb640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:27:02.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.508+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e00004eb0 con 0x7f8e1c13f710 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.537+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3646250369 0 0) 0x7f8e00004eb0 con 0x7f8e1c13f710 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.537+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e1c13c530 con 0x7f8e1c13f710 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.537+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8e0c001ba0 con 0x7f8e1c13f710 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.537+0000 7f8e11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.537+0000 7f8e11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.537+0000 7f8e11ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:02.537 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:02.538 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:02.538 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:02.538 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:02.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.537+0000 7f8e11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:02.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3404731895 0 0) 0x7f8e1c13c530 con 0x7f8e1c13f710 2024-03-25T19:27:02.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 >> v1:172.21.15.60:6790/0 conn(0x7f8e1c1430c0 legacy=0x7f8e1c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 >> v1:172.21.15.90:6789/0 conn(0x7f8e1c0a98f0 legacy=0x7f8e1c0b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 1 monclient: found mon.a 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c131320 con 0x7f8e1c13f710 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e236fb640 5 monclient: authenticate success, global_id 4527 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e236fb640 10 monclient: _renew_subs 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e236fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:02.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.562+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c13c860 con 0x7f8e1c13f710 2024-03-25T19:27:02.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.563+0000 7f8e236fb640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:02.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.563+0000 7f8e236fb640 10 monclient: _renew_subs 2024-03-25T19:27:02.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.563+0000 7f8e236fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:02.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.563+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c13bf20 con 0x7f8e1c13f710 2024-03-25T19:27:02.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.563+0000 7f8e236fb640 1 librados: init done 2024-03-25T19:27:02.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.564+0000 7f8e236fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:02.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.564+0000 7f8e236fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:02.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.564+0000 7f8e236fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:02.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.564+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c0627b0 con 0x7f8e1c13f710 2024-03-25T19:27:02.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.894+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e0c004a90 con 0x7f8e1c13f710 2024-03-25T19:27:02.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.894+0000 7f8e11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:02.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.894+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f8e0c004da0 con 0x7f8e1c13f710 2024-03-25T19:27:02.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.894+0000 7f8e11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:02.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.894+0000 7f8e11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.894+0000 7f8e11ffb640 10 monclient: dump: 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:02.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.894+0000 7f8e137fe640 4 set_mon_vals no callback set 2024-03-25T19:27:02.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.895+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f8e0c01ad70 con 0x7f8e1c13f710 2024-03-25T19:27:02.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.896+0000 7f8e21497640 10 monclient: get_auth_request con 0x7f8e0006dd00 auth_method 0 2024-03-25T19:27:02.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.896+0000 7f8e11ffb640 10 client.4527.objecter ms_handle_connect 0x7f8e0006dd00 2024-03-25T19:27:02.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f8e0c054910 con 0x7f8e1c13f710 2024-03-25T19:27:02.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 10 client.4527.objecter ms_dispatch 0x7f8e1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:02.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:02.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:02.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T19:27:02.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e0c050420 con 0x7f8e1c13f710 2024-03-25T19:27:02.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:02.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:02.916+0000 7f8e11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:03.167 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.167+0000 7f8e236fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:03.167 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.167+0000 7f8e236fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:03.167 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.167+0000 7f8e236fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:03.167 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.167+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e1c13cb60 con 0x7f8e1c13f710 2024-03-25T19:27:03.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.225+0000 7f8e11ffb640 1 -- 172.21.15.60:0/1760030376 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f8e0c052350 con 0x7f8e1c13f710 2024-03-25T19:27:03.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.225+0000 7f8e11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:03.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.225+0000 7f8e11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:03.225 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:03.226 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 10 librados: watch_flush enter 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 10 librados: watch_flush exit 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 10 client.4527.objecter successfully canceled tick 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 >> v1:172.21.15.90:6800/1827638754 conn(0x7f8e0006dd00 legacy=0x7f8e000701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 10 monclient: shutdown 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 >> v1:172.21.15.60:6789/0 conn(0x7f8e1c13f710 legacy=0x7f8e1c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:03.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 shutdown_connections 2024-03-25T19:27:03.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 >> 172.21.15.60:0/1760030376 conn(0x7f8e1c0b7ea0 msgr2=0x7f8e1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:03.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 shutdown_connections 2024-03-25T19:27:03.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 1 -- 172.21.15.60:0/1760030376 wait complete. 2024-03-25T19:27:03.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:03.228+0000 7f8e236fb640 1 librados: shutdown 2024-03-25T19:27:03.241 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:10.047 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:10.162 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.163+0000 7f86c534b640 10 monclient: get_monmap_and_config 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.163+0000 7f86c534b640 10 monclient: build_initial_monmap 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.163+0000 7f86c534b640 10 monclient: monmap: 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:10.164061+0000 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:10.164061+0000 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:10.163 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.164+0000 7f86c534b640 1 Processor -- start 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.164+0000 7f86c534b640 1 -- start start 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.164+0000 7f86c534b640 10 monclient: init 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 10 monclient(hunting): picked mon.noname-b con 0x7f86c00a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 10 monclient(hunting): picked mon.noname-a con 0x7f86c013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 10 monclient(hunting): picked mon.noname-c con 0x7f86c01430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0142520 con 0x7f86c013f710 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00b28a0 con 0x7f86c01430c0 2024-03-25T19:27:10.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00b2a30 con 0x7f86c00a98f0 2024-03-25T19:27:10.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:10.166 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.165+0000 7f86c534b640 10 monclient(hunting): authenticate will time out at 6757.263079s 2024-03-25T19:27:10.170 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.170+0000 7f86be7fc640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f86c013f710 0x7f86c01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:39452/0 (socket says 172.21.15.60:39452) 2024-03-25T19:27:10.170 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.170+0000 7f86be7fc640 1 -- 172.21.15.60:0/2472129385 learned_addr learned my addr 172.21.15.60:0/2472129385 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:10.182 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.183+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222340825 0 0) 0x7f86c00b28a0 con 0x7f86c01430c0 2024-03-25T19:27:10.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.183+0000 7f86bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:10.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.183+0000 7f86bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:10.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.183+0000 7f86bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:10.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.183+0000 7f86bdffb640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:27:10.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.183+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a8003520 con 0x7f86c01430c0 2024-03-25T19:27:10.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.184+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322068717 0 0) 0x7f86c0142520 con 0x7f86c013f710 2024-03-25T19:27:10.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.184+0000 7f86bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:10.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.184+0000 7f86bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:10.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.184+0000 7f86bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:10.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.184+0000 7f86bdffb640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:27:10.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.184+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c00b28a0 con 0x7f86c013f710 2024-03-25T19:27:10.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967871026 0 0) 0x7f86c00b2a30 con 0x7f86c00a98f0 2024-03-25T19:27:10.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:10.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:10.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:10.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:27:10.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0142520 con 0x7f86c00a98f0 2024-03-25T19:27:10.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338578837 0 0) 0x7f86c00b28a0 con 0x7f86c013f710 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 >> v1:172.21.15.60:6790/0 conn(0x7f86c00a98f0 legacy=0x7f86c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 >> v1:172.21.15.90:6789/0 conn(0x7f86c01430c0 legacy=0x7f86c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 monclient: found mon.noname-a 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c00b44c0 con 0x7f86c013f710 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86b4001670 con 0x7f86c013f710 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86bdffb640 10 monclient: dump: 2024-03-25T19:27:10.191 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86c534b640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:27:10.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.190+0000 7f86c534b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:10.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b40017d0 con 0x7f86c013f710 2024-03-25T19:27:10.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86bdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:10.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86bdffb640 1 -- 172.21.15.60:0/2472129385 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86b4001be0 con 0x7f86c013f710 2024-03-25T19:27:10.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86c534b640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:10.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86c534b640 4 set_mon_vals no callback set 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86bdffb640 10 monclient: dump: 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:10.204 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86c534b640 10 monclient: shutdown 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86c534b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.204+0000 7f86c534b640 1 -- 172.21.15.60:0/2472129385 >> v1:172.21.15.60:6789/0 conn(0x7f86c013f710 legacy=0x7f86c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.205+0000 7f86c534b640 1 -- 172.21.15.60:0/2472129385 shutdown_connections 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.205+0000 7f86c534b640 1 -- 172.21.15.60:0/2472129385 >> 172.21.15.60:0/2472129385 conn(0x7f86c00b7ea0 msgr2=0x7f86c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.205+0000 7f86c534b640 1 -- 172.21.15.60:0/2472129385 shutdown_connections 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.205+0000 7f86c534b640 1 -- 172.21.15.60:0/2472129385 wait complete. 2024-03-25T19:27:10.206 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.205+0000 7f86c534b640 10 monclient: build_initial_monmap 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.205+0000 7f86c534b640 10 monclient: monmap: 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.206+0000 7f86c534b640 1 librados: starting msgr at 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.206+0000 7f86c534b640 1 librados: starting objecter 2024-03-25T19:27:10.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.206+0000 7f86c534b640 1 Processor -- start 2024-03-25T19:27:10.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.206+0000 7f86c534b640 1 -- start start 2024-03-25T19:27:10.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.206+0000 7f86c534b640 1 librados: setting wanted keys 2024-03-25T19:27:10.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.206+0000 7f86c534b640 1 librados: calling monclient init 2024-03-25T19:27:10.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.206+0000 7f86c534b640 10 monclient: init 2024-03-25T19:27:10.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:10.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:10.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 10 monclient(hunting): picked mon.noname-b con 0x7f86c00a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:10.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 10 monclient(hunting): picked mon.noname-a con 0x7f86c013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:10.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 10 monclient(hunting): picked mon.noname-c con 0x7f86c01430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:10.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0138740 con 0x7f86c013f710 2024-03-25T19:27:10.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0139970 con 0x7f86c00a98f0 2024-03-25T19:27:10.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0138070 con 0x7f86c01430c0 2024-03-25T19:27:10.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:10.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.207+0000 7f86c534b640 10 monclient(hunting): authenticate will time out at 6757.305648s 2024-03-25T19:27:10.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.213+0000 7f86beffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f86c00a98f0 0x7f86c0137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:45544/0 (socket says 172.21.15.60:45544) 2024-03-25T19:27:10.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.213+0000 7f86beffd640 1 -- 172.21.15.60:0/520816821 learned_addr learned my addr 172.21.15.60:0/520816821 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:10.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.214+0000 7f86a77fe640 10 client.?.objecter ms_handle_connect 0x7f86c00a98f0 2024-03-25T19:27:10.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.214+0000 7f86a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:10.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.222+0000 7f86a77fe640 10 client.?.objecter ms_handle_connect 0x7f86c01430c0 2024-03-25T19:27:10.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.222+0000 7f86a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:10.222 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.223+0000 7f86a77fe640 10 client.?.objecter ms_handle_connect 0x7f86c013f710 2024-03-25T19:27:10.222 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.223+0000 7f86a77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:10.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.232+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598739942 0 0) 0x7f86c0138070 con 0x7f86c01430c0 2024-03-25T19:27:10.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.232+0000 7f86a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:27:10.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.232+0000 7f86a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:10.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.232+0000 7f86a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:10.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.232+0000 7f86a77fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:27:10.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.232+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ac004eb0 con 0x7f86c01430c0 2024-03-25T19:27:10.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.243+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237110410 0 0) 0x7f86c0139970 con 0x7f86c00a98f0 2024-03-25T19:27:10.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.243+0000 7f86a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:27:10.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.243+0000 7f86a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:10.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.243+0000 7f86a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:10.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.243+0000 7f86a77fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:27:10.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.243+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0138070 con 0x7f86c00a98f0 2024-03-25T19:27:10.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.245+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293179101 0 0) 0x7f86c0138740 con 0x7f86c013f710 2024-03-25T19:27:10.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.245+0000 7f86a77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:27:10.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.245+0000 7f86a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:10.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.245+0000 7f86a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:10.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.245+0000 7f86a77fe640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:27:10.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.245+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c0139970 con 0x7f86c013f710 2024-03-25T19:27:10.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.252+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675580092 0 0) 0x7f86ac004eb0 con 0x7f86c01430c0 2024-03-25T19:27:10.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.252+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c0138740 con 0x7f86c01430c0 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.252+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86b8001af0 con 0x7f86c01430c0 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.252+0000 7f86a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.252+0000 7f86a77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.252+0000 7f86a77fe640 10 monclient(hunting): dump: 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:10.252 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:10.253 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:10.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.252+0000 7f86a77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:10.266 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 435067069 0 0) 0x7f86c0138070 con 0x7f86c00a98f0 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86ac004eb0 con 0x7f86c00a98f0 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86b0001a80 con 0x7f86c00a98f0 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 10 monclient(hunting): dump: 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:10.267 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4272969073 0 0) 0x7f86c0138740 con 0x7f86c01430c0 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 >> v1:172.21.15.90:6789/0 conn(0x7f86c00a98f0 legacy=0x7f86c0137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 >> v1:172.21.15.60:6789/0 conn(0x7f86c013f710 legacy=0x7f86c01da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 monclient: found mon.c 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0138200 con 0x7f86c01430c0 2024-03-25T19:27:10.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 10 monclient: _finish_auth 0 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 5 monclient: authenticate success, global_id 4523 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 10 monclient: _renew_subs 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c0138470 con 0x7f86c01430c0 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 10 monclient: _renew_subs 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c01db1e0 con 0x7f86c01430c0 2024-03-25T19:27:10.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.266+0000 7f86c534b640 1 librados: init done 2024-03-25T19:27:10.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.267+0000 7f86c534b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:10.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.267+0000 7f86c534b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:10.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.267+0000 7f86c534b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:27:10.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.267+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c00627b0 con 0x7f86c01430c0 2024-03-25T19:27:10.572 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.572+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86b8001fe0 con 0x7f86c01430c0 2024-03-25T19:27:10.572 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.572+0000 7f86a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:10.572 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.572+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f86b8003b30 con 0x7f86c01430c0 2024-03-25T19:27:10.572 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.572+0000 7f86a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:10.572 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.572+0000 7f86a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.572+0000 7f86a77fe640 10 monclient: dump: 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:10.573 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.572+0000 7f86bd7fa640 4 set_mon_vals no callback set 2024-03-25T19:27:10.574 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.573+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f86b8019b00 con 0x7f86c01430c0 2024-03-25T19:27:10.574 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.574+0000 7f86beffd640 10 monclient: get_auth_request con 0x7f86ac06df20 auth_method 0 2024-03-25T19:27:10.574 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.574+0000 7f86a77fe640 10 client.4523.objecter ms_handle_connect 0x7f86ac06df20 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f86b8053860 con 0x7f86c01430c0 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 10 client.4523.objecter ms_dispatch 0x7f86c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86b804f1a0 con 0x7f86c01430c0 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:10.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.582+0000 7f86a77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:10.828 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.828+0000 7f86c534b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:10.828 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.828+0000 7f86c534b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:10.828 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.828+0000 7f86c534b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:27:10.828 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.828+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86c0074cc0 con 0x7f86c01430c0 2024-03-25T19:27:10.940 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.940+0000 7f86a77fe640 1 -- 172.21.15.60:0/520816821 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f86b80512a0 con 0x7f86c01430c0 2024-03-25T19:27:10.940 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.940+0000 7f86a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:10.940 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.940+0000 7f86a77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:10.940 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:10.940 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:10.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 10 librados: watch_flush enter 2024-03-25T19:27:10.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 10 librados: watch_flush exit 2024-03-25T19:27:10.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T19:27:10.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 10 client.4523.objecter successfully canceled tick 2024-03-25T19:27:10.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 >> v1:172.21.15.90:6800/1827638754 conn(0x7f86ac06df20 legacy=0x7f86ac0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:10.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 10 monclient: shutdown 2024-03-25T19:27:10.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:10.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 >> v1:172.21.15.60:6790/0 conn(0x7f86c01430c0 legacy=0x7f86c01e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:10.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 shutdown_connections 2024-03-25T19:27:10.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 >> 172.21.15.60:0/520816821 conn(0x7f86c00b7ea0 msgr2=0x7f86c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:10.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 shutdown_connections 2024-03-25T19:27:10.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 1 -- 172.21.15.60:0/520816821 wait complete. 2024-03-25T19:27:10.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:10.943+0000 7f86c534b640 1 librados: shutdown 2024-03-25T19:27:10.953 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:17.760 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:17.875 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.876+0000 7fd51b2d0640 10 monclient: get_monmap_and_config 2024-03-25T19:27:17.875 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.876+0000 7fd51b2d0640 10 monclient: build_initial_monmap 2024-03-25T19:27:17.875 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.876+0000 7fd51b2d0640 10 monclient: monmap: 2024-03-25T19:27:17.875 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:17.875 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:17.875 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:17.876769+0000 2024-03-25T19:27:17.875 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:17.876769+0000 2024-03-25T19:27:17.876 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:17.876 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:17.876 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:17.876 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:17.876 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:17.876 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.876+0000 7fd51b2d0640 1 Processor -- start 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 1 -- start start 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient: init 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd5140b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd5140b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd51414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5140af940 con 0x7fd51414be90 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5140b0a70 con 0x7fd5140b4d60 2024-03-25T19:27:17.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5140b0d50 con 0x7fd5140b1730 2024-03-25T19:27:17.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:17.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.877+0000 7fd51b2d0640 10 monclient(hunting): authenticate will time out at 6764.975775s 2024-03-25T19:27:17.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.882+0000 7fd51906c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd5140b4d60 0x7fd51413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:58970/0 (socket says 172.21.15.60:58970) 2024-03-25T19:27:17.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.882+0000 7fd51906c640 1 -- 172.21.15.60:0/616493730 learned_addr learned my addr 172.21.15.60:0/616493730 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:17.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531134907 0 0) 0x7fd5140b0a70 con 0x7fd5140b4d60 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd500003520 con 0x7fd5140b4d60 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353362664 0 0) 0x7fd5140b0d50 con 0x7fd5140b1730 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:17.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:17.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:27:17.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.902+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5140b0a70 con 0x7fd5140b1730 2024-03-25T19:27:17.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.908+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645471270 0 0) 0x7fd5140af940 con 0x7fd51414be90 2024-03-25T19:27:17.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.908+0000 7fd50bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:17.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.908+0000 7fd50bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:17.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.908+0000 7fd50bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:17.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.908+0000 7fd50bfff640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:27:17.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.908+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5140b0d50 con 0x7fd51414be90 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383197984 0 0) 0x7fd5140b0a70 con 0x7fd5140b1730 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 >> v1:172.21.15.90:6789/0 conn(0x7fd5140b4d60 legacy=0x7fd51413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 >> v1:172.21.15.60:6789/0 conn(0x7fd51414be90 legacy=0x7fd51414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 1 monclient: found mon.noname-b 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5140b1030 con 0x7fd5140b1730 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 10 monclient: _finish_auth 0 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:17.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd504001670 con 0x7fd5140b1730 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd50bfff640 10 monclient: dump: 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:17.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd51b2d0640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:27:17.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.911+0000 7fd51b2d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:17.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5040017d0 con 0x7fd5140b1730 2024-03-25T19:27:17.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd50bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:17.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd50bfff640 1 -- 172.21.15.60:0/616493730 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd504001be0 con 0x7fd5140b1730 2024-03-25T19:27:17.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd51b2d0640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:17.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd51b2d0640 4 set_mon_vals no callback set 2024-03-25T19:27:17.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd50bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd50bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd50bfff640 10 monclient: dump: 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:17.926 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:17.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd51b2d0640 10 monclient: shutdown 2024-03-25T19:27:17.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd51b2d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:17.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd51b2d0640 1 -- 172.21.15.60:0/616493730 >> v1:172.21.15.60:6790/0 conn(0x7fd5140b1730 legacy=0x7fd5140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:17.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd51b2d0640 1 -- 172.21.15.60:0/616493730 shutdown_connections 2024-03-25T19:27:17.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.926+0000 7fd51b2d0640 1 -- 172.21.15.60:0/616493730 >> 172.21.15.60:0/616493730 conn(0x7fd5140617a0 msgr2=0x7fd514061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:17.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.927+0000 7fd51b2d0640 1 -- 172.21.15.60:0/616493730 shutdown_connections 2024-03-25T19:27:17.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.927+0000 7fd51b2d0640 1 -- 172.21.15.60:0/616493730 wait complete. 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.927+0000 7fd51b2d0640 10 monclient: build_initial_monmap 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.927+0000 7fd51b2d0640 10 monclient: monmap: 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 librados: starting msgr at 2024-03-25T19:27:17.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 librados: starting objecter 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 Processor -- start 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 -- start start 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 librados: setting wanted keys 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 librados: calling monclient init 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient: init 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient(hunting): picked mon.noname-b con 0x7fd5140b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient(hunting): picked mon.noname-a con 0x7fd5140b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient(hunting): picked mon.noname-c con 0x7fd51414be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd514136d70 con 0x7fd5140b4d60 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5141306a0 con 0x7fd5140b1730 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd514135b00 con 0x7fd51414be90 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:17.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.928+0000 7fd51b2d0640 10 monclient(hunting): authenticate will time out at 6765.026815s 2024-03-25T19:27:17.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.936+0000 7fd51906c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fd5140b1730 0x7fd51412ff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:59000/0 (socket says 172.21.15.60:59000) 2024-03-25T19:27:17.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.936+0000 7fd51906c640 1 -- 172.21.15.60:0/4196689616 learned_addr learned my addr 172.21.15.60:0/4196689616 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:17.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.939+0000 7fd509ffb640 10 client.?.objecter ms_handle_connect 0x7fd5140b1730 2024-03-25T19:27:17.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.939+0000 7fd509ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:17.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.948+0000 7fd509ffb640 10 client.?.objecter ms_handle_connect 0x7fd5140b4d60 2024-03-25T19:27:17.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.948+0000 7fd509ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:17.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.948+0000 7fd509ffb640 10 client.?.objecter ms_handle_connect 0x7fd51414be90 2024-03-25T19:27:17.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.948+0000 7fd509ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:17.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.953+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943419400 0 0) 0x7fd5141306a0 con 0x7fd5140b1730 2024-03-25T19:27:17.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.953+0000 7fd509ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:17.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.953+0000 7fd509ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:17.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.953+0000 7fd509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:17.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.953+0000 7fd509ffb640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:27:17.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.953+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f8004eb0 con 0x7fd5140b1730 2024-03-25T19:27:17.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.958+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858604008 0 0) 0x7fd514136d70 con 0x7fd5140b4d60 2024-03-25T19:27:17.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.958+0000 7fd509ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:17.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.958+0000 7fd509ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:17.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.958+0000 7fd509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:17.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.958+0000 7fd509ffb640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:27:17.958 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.958+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5141306a0 con 0x7fd5140b4d60 2024-03-25T19:27:17.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.966+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236947372 0 0) 0x7fd514135b00 con 0x7fd51414be90 2024-03-25T19:27:17.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.966+0000 7fd509ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:17.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.966+0000 7fd509ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:17.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.966+0000 7fd509ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:17.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.966+0000 7fd509ffb640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:27:17.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.966+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd514136d70 con 0x7fd51414be90 2024-03-25T19:27:17.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.968+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3035415239 0 0) 0x7fd4f8004eb0 con 0x7fd5140b1730 2024-03-25T19:27:17.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.968+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd514135b00 con 0x7fd5140b1730 2024-03-25T19:27:17.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.968+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd510001950 con 0x7fd5140b1730 2024-03-25T19:27:17.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.968+0000 7fd509ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:17.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.968+0000 7fd509ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:17.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.968+0000 7fd509ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:17.968 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:17.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.968+0000 7fd509ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2361170421 0 0) 0x7fd514135b00 con 0x7fd5140b1730 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 >> v1:172.21.15.60:6790/0 conn(0x7fd51414be90 legacy=0x7fd5141d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 >> v1:172.21.15.60:6789/0 conn(0x7fd5140b4d60 legacy=0x7fd5141d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 1 monclient: found mon.b 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd514135c90 con 0x7fd5140b1730 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:17.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd509ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 10 monclient: _renew_subs 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51413e930 con 0x7fd5140b1730 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 10 monclient: _renew_subs 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd514145500 con 0x7fd5140b1730 2024-03-25T19:27:17.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.979+0000 7fd51b2d0640 1 librados: init done 2024-03-25T19:27:17.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.980+0000 7fd51b2d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:17.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.980+0000 7fd51b2d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:17.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.980+0000 7fd51b2d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:17.981 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:17.980+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5140627b0 con 0x7fd5140b1730 2024-03-25T19:27:18.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.254+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5100047e0 con 0x7fd5140b1730 2024-03-25T19:27:18.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.254+0000 7fd509ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:18.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.254+0000 7fd50b7fe640 4 set_mon_vals no callback set 2024-03-25T19:27:18.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.254+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fd510001d80 con 0x7fd5140b1730 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.254+0000 7fd509ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.254+0000 7fd509ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.254+0000 7fd509ffb640 10 monclient: dump: 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:18.255 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:18.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.255+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fd510004c30 con 0x7fd5140b1730 2024-03-25T19:27:18.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.255+0000 7fd51886b640 10 monclient: get_auth_request con 0x7fd4f806ddf0 auth_method 0 2024-03-25T19:27:18.256 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.256+0000 7fd509ffb640 10 client.4531.objecter ms_handle_connect 0x7fd4f806ddf0 2024-03-25T19:27:18.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.261+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fd510054890 con 0x7fd5140b1730 2024-03-25T19:27:18.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.261+0000 7fd509ffb640 10 client.4531.objecter ms_dispatch 0x7fd514000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:18.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.261+0000 7fd509ffb640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:18.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.261+0000 7fd509ffb640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:18.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.261+0000 7fd509ffb640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T19:27:18.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.262+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5100501d0 con 0x7fd5140b1730 2024-03-25T19:27:18.262 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.262+0000 7fd509ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:18.262 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.262+0000 7fd509ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:18.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.525+0000 7fd51b2d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:18.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.525+0000 7fd51b2d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:18.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.525+0000 7fd51b2d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:18.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.525+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5141361b0 con 0x7fd5140b1730 2024-03-25T19:27:18.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.552+0000 7fd509ffb640 1 -- 172.21.15.60:0/4196689616 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fd5100522d0 con 0x7fd5140b1730 2024-03-25T19:27:18.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.553+0000 7fd509ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:18.552 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.553+0000 7fd509ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:18.552 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:18.553 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:18.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 10 librados: watch_flush enter 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 10 librados: watch_flush exit 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 10 client.4531.objecter successfully canceled tick 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 >> v1:172.21.15.90:6800/1827638754 conn(0x7fd4f806ddf0 legacy=0x7fd4f80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 10 monclient: shutdown 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.555+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 >> v1:172.21.15.90:6789/0 conn(0x7fd5140b1730 legacy=0x7fd51412ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.556+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 shutdown_connections 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.556+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 >> 172.21.15.60:0/4196689616 conn(0x7fd5140617a0 msgr2=0x7fd5140b6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.556+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 shutdown_connections 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.556+0000 7fd51b2d0640 1 -- 172.21.15.60:0/4196689616 wait complete. 2024-03-25T19:27:18.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:18.556+0000 7fd51b2d0640 1 librados: shutdown 2024-03-25T19:27:18.567 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:20.261 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:20.915 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.262 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:20.973 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.264 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:20.918 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.277 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:20.988 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.299 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:20.953 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.313 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:21.024 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.337 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:21.048 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.342 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:21.053 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:20.374 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:21.086 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:25.374 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:25.495 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.495+0000 7ff372463640 10 monclient: get_monmap_and_config 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.495+0000 7ff372463640 10 monclient: build_initial_monmap 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.496+0000 7ff372463640 10 monclient: monmap: 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:25.496689+0000 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:25.496689+0000 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:25.496 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:25.497 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:25.497 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.496+0000 7ff372463640 1 Processor -- start 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.496+0000 7ff372463640 1 -- start start 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.496+0000 7ff372463640 10 monclient: init 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 10 monclient(hunting): picked mon.noname-a con 0x7ff36c0b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 10 monclient(hunting): picked mon.noname-b con 0x7ff36c0b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 10 monclient(hunting): picked mon.noname-c con 0x7ff36c14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c0af940 con 0x7ff36c0b4d60 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c0b0a70 con 0x7ff36c14be90 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c0b0d50 con 0x7ff36c0b1730 2024-03-25T19:27:25.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:25.499 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.497+0000 7ff372463640 10 monclient(hunting): authenticate will time out at 6772.595529s 2024-03-25T19:27:25.510 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.511+0000 7ff370a00640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff36c14be90 0x7ff36c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:35336/0 (socket says 172.21.15.60:35336) 2024-03-25T19:27:25.510 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.511+0000 7ff370a00640 1 -- 172.21.15.60:0/2497625161 learned_addr learned my addr 172.21.15.60:0/2497625161 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:25.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433285675 0 0) 0x7ff36c0b0a70 con 0x7ff36c14be90 2024-03-25T19:27:25.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:25.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:25.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:25.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:27:25.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff358003520 con 0x7ff36c14be90 2024-03-25T19:27:25.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922091444 0 0) 0x7ff36c0af940 con 0x7ff36c0b4d60 2024-03-25T19:27:25.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:25.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:25.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff36c0b0a70 con 0x7ff36c0b4d60 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848599235 0 0) 0x7ff36c0b0d50 con 0x7ff36c0b1730 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:27:25.539 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.538+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff36c0af940 con 0x7ff36c0b1730 2024-03-25T19:27:25.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.549+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928491267 0 0) 0x7ff36c0af940 con 0x7ff36c0b1730 2024-03-25T19:27:25.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.549+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 >> v1:172.21.15.90:6789/0 conn(0x7ff36c14be90 legacy=0x7ff36c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 >> v1:172.21.15.60:6789/0 conn(0x7ff36c0b4d60 legacy=0x7ff36c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 1 monclient: found mon.noname-b 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c0b1030 con 0x7ff36c0b1730 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 10 monclient: _finish_auth 0 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff3540015b0 con 0x7ff36c0b1730 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:25.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff36affd640 10 monclient: dump: 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff372463640 5 monclient: authenticate success, global_id 4532 2024-03-25T19:27:25.551 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.550+0000 7ff372463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff354001710 con 0x7ff36c0b1730 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff36affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff36affd640 1 -- 172.21.15.60:0/2497625161 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff354001b60 con 0x7ff36c0b1730 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff36affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff36affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff36affd640 10 monclient: dump: 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.561 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:25.562 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:25.562 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:25.562 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:25.562 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:25.562 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:25.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff372463640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:25.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.561+0000 7ff372463640 4 set_mon_vals no callback set 2024-03-25T19:27:25.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.562+0000 7ff372463640 10 monclient: shutdown 2024-03-25T19:27:25.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.562+0000 7ff372463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:25.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.562+0000 7ff372463640 1 -- 172.21.15.60:0/2497625161 >> v1:172.21.15.60:6790/0 conn(0x7ff36c0b1730 legacy=0x7ff36c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.562+0000 7ff372463640 1 -- 172.21.15.60:0/2497625161 shutdown_connections 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.562+0000 7ff372463640 1 -- 172.21.15.60:0/2497625161 >> 172.21.15.60:0/2497625161 conn(0x7ff36c0617a0 msgr2=0x7ff36c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.562+0000 7ff372463640 1 -- 172.21.15.60:0/2497625161 shutdown_connections 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.562+0000 7ff372463640 1 -- 172.21.15.60:0/2497625161 wait complete. 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.563+0000 7ff372463640 10 monclient: build_initial_monmap 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.563+0000 7ff372463640 10 monclient: monmap: 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:25.565 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.563+0000 7ff372463640 1 librados: starting msgr at 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.563+0000 7ff372463640 1 librados: starting objecter 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.563+0000 7ff372463640 1 Processor -- start 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 1 -- start start 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 1 librados: setting wanted keys 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 1 librados: calling monclient init 2024-03-25T19:27:25.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient: init 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient(hunting): picked mon.noname-a con 0x7ff36c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient(hunting): picked mon.noname-b con 0x7ff36c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient(hunting): picked mon.noname-c con 0x7ff36c14be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c1366a0 con 0x7ff36c0b1730 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c137d60 con 0x7ff36c0b4d60 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c136b20 con 0x7ff36c14be90 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:25.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.564+0000 7ff372463640 10 monclient(hunting): authenticate will time out at 6772.662682s 2024-03-25T19:27:25.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.577+0000 7ff36b7fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff36c0b4d60 0x7ff36c1d80b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:35378/0 (socket says 172.21.15.60:35378) 2024-03-25T19:27:25.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.577+0000 7ff36b7fe640 1 -- 172.21.15.60:0/404088124 learned_addr learned my addr 172.21.15.60:0/404088124 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:25.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.579+0000 7ff368ff9640 10 client.?.objecter ms_handle_connect 0x7ff36c0b4d60 2024-03-25T19:27:25.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.579+0000 7ff368ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:25.583 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.584+0000 7ff368ff9640 10 client.?.objecter ms_handle_connect 0x7ff36c14be90 2024-03-25T19:27:25.584 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.584+0000 7ff368ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:25.590 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.590+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934692162 0 0) 0x7ff36c137d60 con 0x7ff36c0b4d60 2024-03-25T19:27:25.590 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.590+0000 7ff368ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:25.590 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.590+0000 7ff368ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:25.590 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.590+0000 7ff368ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:25.590 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.590+0000 7ff368ff9640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:27:25.590 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.590+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff34c004eb0 con 0x7ff36c0b4d60 2024-03-25T19:27:25.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.595+0000 7ff368ff9640 10 client.?.objecter ms_handle_connect 0x7ff36c0b1730 2024-03-25T19:27:25.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.595+0000 7ff368ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:25.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.599+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884403728 0 0) 0x7ff36c1366a0 con 0x7ff36c0b1730 2024-03-25T19:27:25.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.599+0000 7ff368ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:25.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.599+0000 7ff368ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:25.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.599+0000 7ff368ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:25.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.599+0000 7ff368ff9640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:27:25.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.599+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff36c137d60 con 0x7ff36c0b1730 2024-03-25T19:27:25.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.608+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2288871560 0 0) 0x7ff34c004eb0 con 0x7ff36c0b4d60 2024-03-25T19:27:25.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.608+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff36c1366a0 con 0x7ff36c0b4d60 2024-03-25T19:27:25.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.608+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff354001b20 con 0x7ff36c0b4d60 2024-03-25T19:27:25.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.608+0000 7ff368ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:25.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.608+0000 7ff368ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:25.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.608+0000 7ff368ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:25.608 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:25.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.608+0000 7ff368ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.637+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1448788709 0 0) 0x7ff36c137d60 con 0x7ff36c0b1730 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.637+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff34c004eb0 con 0x7ff36c0b1730 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.637+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff364001930 con 0x7ff36c0b1730 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.637+0000 7ff368ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.637+0000 7ff368ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.637+0000 7ff368ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:25.637 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:25.638 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:25.638 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:25.638 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:25.638 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1563641701 0 0) 0x7ff34c004eb0 con 0x7ff36c0b1730 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 >> v1:172.21.15.60:6790/0 conn(0x7ff36c14be90 legacy=0x7ff36c1e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 >> v1:172.21.15.90:6789/0 conn(0x7ff36c0b4d60 legacy=0x7ff36c1d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 1 monclient: found mon.a 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c136cb0 con 0x7ff36c0b1730 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 10 monclient: _finish_auth 0 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:27:25.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff368ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 5 monclient: authenticate success, global_id 4545 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 10 monclient: _renew_subs 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c136010 con 0x7ff36c0b1730 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 10 monclient: _renew_subs 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36c13ec30 con 0x7ff36c0b1730 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.696+0000 7ff372463640 1 librados: init done 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.697+0000 7ff372463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.697+0000 7ff372463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.697+0000 7ff372463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:25.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:25.697+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36c1401f0 con 0x7ff36c0b1730 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff364003610 con 0x7ff36c0b1730 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff368ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff3640039a0 con 0x7ff36c0b1730 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff368ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff368ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff368ff9640 10 monclient: dump: 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:26.023 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff36a7fc640 4 set_mon_vals no callback set 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.023+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7ff364003bc0 con 0x7ff36c0b1730 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.024+0000 7ff36b7fe640 10 monclient: get_auth_request con 0x7ff34c06df20 auth_method 0 2024-03-25T19:27:26.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.024+0000 7ff368ff9640 10 client.4545.objecter ms_handle_connect 0x7ff34c06df20 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7ff364053590 con 0x7ff36c0b1730 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 10 client.4545.objecter ms_dispatch 0x7ff36c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff36404f0a0 con 0x7ff36c0b1730 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:26.030 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.030+0000 7ff368ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:26.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.279+0000 7ff372463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:26.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.279+0000 7ff372463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:26.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.279+0000 7ff372463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:26.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.279+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff36c0627b0 con 0x7ff36c0b1730 2024-03-25T19:27:26.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.317+0000 7ff368ff9640 1 -- 172.21.15.60:0/404088124 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7ff364050fd0 con 0x7ff36c0b1730 2024-03-25T19:27:26.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.317+0000 7ff368ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:26.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.317+0000 7ff368ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:26.317 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:26.318 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:26.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 10 librados: watch_flush enter 2024-03-25T19:27:26.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 10 librados: watch_flush exit 2024-03-25T19:27:26.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T19:27:26.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 10 client.4545.objecter successfully canceled tick 2024-03-25T19:27:26.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 >> v1:172.21.15.90:6800/1827638754 conn(0x7ff34c06df20 legacy=0x7ff34c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 10 monclient: shutdown 2024-03-25T19:27:26.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:26.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 >> v1:172.21.15.60:6789/0 conn(0x7ff36c0b1730 legacy=0x7ff36c1d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:26.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 shutdown_connections 2024-03-25T19:27:26.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 >> 172.21.15.60:0/404088124 conn(0x7ff36c0617a0 msgr2=0x7ff36c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:26.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 shutdown_connections 2024-03-25T19:27:26.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 1 -- 172.21.15.60:0/404088124 wait complete. 2024-03-25T19:27:26.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:26.320+0000 7ff372463640 1 librados: shutdown 2024-03-25T19:27:26.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:33.138 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:33.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.258+0000 7f1127c88640 10 monclient: get_monmap_and_config 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.258+0000 7f1127c88640 10 monclient: build_initial_monmap 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.258+0000 7f1127c88640 10 monclient: monmap: 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:33.259013+0000 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:33.259013+0000 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:33.258 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.259+0000 7f1127c88640 1 Processor -- start 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.259+0000 7f1127c88640 1 -- start start 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.259+0000 7f1127c88640 10 monclient: init 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.259+0000 7f1127c88640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.259+0000 7f1127c88640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 10 monclient(hunting): picked mon.noname-b con 0x7f11200a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 10 monclient(hunting): picked mon.noname-a con 0x7f112013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 10 monclient(hunting): picked mon.noname-c con 0x7f11201430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1120142520 con 0x7f112013f710 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11200b28a0 con 0x7f11201430c0 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11200b2a30 con 0x7f11200a98f0 2024-03-25T19:27:33.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:33.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.260+0000 7f1127c88640 10 monclient(hunting): authenticate will time out at 6780.358251s 2024-03-25T19:27:33.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.269+0000 7f1125a24640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f11200a98f0 0x7f11200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:36562/0 (socket says 172.21.15.60:36562) 2024-03-25T19:27:33.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.269+0000 7f1125a24640 1 -- 172.21.15.60:0/1910987437 learned_addr learned my addr 172.21.15.60:0/1910987437 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:33.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.286+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340363648 0 0) 0x7f11200b2a30 con 0x7f11200a98f0 2024-03-25T19:27:33.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.287+0000 7f1124a22640 10 monclient(hunting): _negotiate 2024-03-25T19:27:33.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.287+0000 7f1124a22640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:33.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.287+0000 7f1124a22640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:33.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.287+0000 7f1124a22640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:27:33.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.287+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f110c003520 con 0x7f11200a98f0 2024-03-25T19:27:33.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.290+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029820816 0 0) 0x7f1120142520 con 0x7f112013f710 2024-03-25T19:27:33.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.290+0000 7f1124a22640 10 monclient(hunting): _negotiate 2024-03-25T19:27:33.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.290+0000 7f1124a22640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:33.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.290+0000 7f1124a22640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:33.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.290+0000 7f1124a22640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:27:33.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.290+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11200b2a30 con 0x7f112013f710 2024-03-25T19:27:33.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.293+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004342199 0 0) 0x7f11200b28a0 con 0x7f11201430c0 2024-03-25T19:27:33.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.293+0000 7f1124a22640 10 monclient(hunting): _negotiate 2024-03-25T19:27:33.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.293+0000 7f1124a22640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:33.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.293+0000 7f1124a22640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:33.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.293+0000 7f1124a22640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:27:33.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.293+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120142520 con 0x7f11201430c0 2024-03-25T19:27:33.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104546904 0 0) 0x7f110c003520 con 0x7f11200a98f0 2024-03-25T19:27:33.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 >> v1:172.21.15.90:6789/0 conn(0x7f11201430c0 legacy=0x7f112014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:33.304 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 >> v1:172.21.15.60:6789/0 conn(0x7f112013f710 legacy=0x7f11201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 10 monclient: _finish_hunting 0 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 1 monclient: found mon.noname-b 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11200b44c0 con 0x7f11200a98f0 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 10 monclient: _finish_auth 0 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 10 monclient: _check_auth_tickets 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f111c001690 con 0x7f11200a98f0 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1127c88640 5 monclient: authenticate success, global_id 4538 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1127c88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:33.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1124a22640 10 monclient: dump: 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:33.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.304+0000 7f1127c88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:33.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f111c0017f0 con 0x7f11200a98f0 2024-03-25T19:27:33.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1124a22640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:33.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1124a22640 1 -- 172.21.15.60:0/1910987437 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f111c001c00 con 0x7f11200a98f0 2024-03-25T19:27:33.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1124a22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:33.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1124a22640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:33.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1124a22640 10 monclient: dump: 2024-03-25T19:27:33.322 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1127c88640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:33.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.322+0000 7f1127c88640 4 set_mon_vals no callback set 2024-03-25T19:27:33.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.323+0000 7f1127c88640 10 monclient: shutdown 2024-03-25T19:27:33.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.323+0000 7f1127c88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:33.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.323+0000 7f1127c88640 1 -- 172.21.15.60:0/1910987437 >> v1:172.21.15.60:6790/0 conn(0x7f11200a98f0 legacy=0x7f11200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.323+0000 7f1127c88640 1 -- 172.21.15.60:0/1910987437 shutdown_connections 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.323+0000 7f1127c88640 1 -- 172.21.15.60:0/1910987437 >> 172.21.15.60:0/1910987437 conn(0x7f11200b7ea0 msgr2=0x7f11200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.323+0000 7f1127c88640 1 -- 172.21.15.60:0/1910987437 shutdown_connections 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.323+0000 7f1127c88640 1 -- 172.21.15.60:0/1910987437 wait complete. 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.324+0000 7f1127c88640 10 monclient: build_initial_monmap 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.324+0000 7f1127c88640 10 monclient: monmap: 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:33.325 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 1 librados: starting msgr at 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 1 librados: starting objecter 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 1 Processor -- start 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 1 -- start start 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 1 librados: setting wanted keys 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 1 librados: calling monclient init 2024-03-25T19:27:33.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 10 monclient: init 2024-03-25T19:27:33.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:33.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:27:33.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 10 monclient(hunting): picked mon.noname-b con 0x7f11200a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:33.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.325+0000 7f1127c88640 10 monclient(hunting): picked mon.noname-a con 0x7f112013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:33.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.326+0000 7f1127c88640 10 monclient(hunting): picked mon.noname-c con 0x7f11201430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:33.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.326+0000 7f1127c88640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11201dc6d0 con 0x7f112013f710 2024-03-25T19:27:33.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.326+0000 7f1127c88640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11201dbaa0 con 0x7f11200a98f0 2024-03-25T19:27:33.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.326+0000 7f1127c88640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112006dea0 con 0x7f11201430c0 2024-03-25T19:27:33.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.326+0000 7f1127c88640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:33.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.326+0000 7f1127c88640 10 monclient(hunting): authenticate will time out at 6780.423924s 2024-03-25T19:27:33.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.332+0000 7f1125a24640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f11200a98f0 0x7f11201dbfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:42370/0 (socket says 172.21.15.60:42370) 2024-03-25T19:27:33.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.332+0000 7f1125a24640 1 -- 172.21.15.60:0/1526787715 learned_addr learned my addr 172.21.15.60:0/1526787715 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:33.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.332+0000 7f11167fc640 10 client.?.objecter ms_handle_connect 0x7f11200a98f0 2024-03-25T19:27:33.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.332+0000 7f11167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:33.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.342+0000 7f11167fc640 10 client.?.objecter ms_handle_connect 0x7f11201430c0 2024-03-25T19:27:33.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.342+0000 7f11167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:33.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.344+0000 7f11167fc640 10 client.?.objecter ms_handle_connect 0x7f112013f710 2024-03-25T19:27:33.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.344+0000 7f11167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:33.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.349+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220941855 0 0) 0x7f11201dbaa0 con 0x7f11200a98f0 2024-03-25T19:27:33.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.349+0000 7f11167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:33.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.349+0000 7f11167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:33.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.349+0000 7f11167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:33.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.349+0000 7f11167fc640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:27:33.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.349+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1104004eb0 con 0x7f11200a98f0 2024-03-25T19:27:33.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.360+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300246921 0 0) 0x7f112006dea0 con 0x7f11201430c0 2024-03-25T19:27:33.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.360+0000 7f11167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:33.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.360+0000 7f11167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:33.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.360+0000 7f11167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:33.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.360+0000 7f11167fc640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:27:33.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.360+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11201dbaa0 con 0x7f11201430c0 2024-03-25T19:27:33.373 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.374+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779986194 0 0) 0x7f11201dc6d0 con 0x7f112013f710 2024-03-25T19:27:33.374 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.374+0000 7f11167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:27:33.374 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.374+0000 7f11167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:33.374 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.374+0000 7f11167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:33.374 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.374+0000 7f11167fc640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:27:33.374 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.374+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112006dea0 con 0x7f112013f710 2024-03-25T19:27:33.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.384+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350442447 0 0) 0x7f1104004eb0 con 0x7f11200a98f0 2024-03-25T19:27:33.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.384+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11201dc6d0 con 0x7f11200a98f0 2024-03-25T19:27:33.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.384+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f111c001b40 con 0x7f11200a98f0 2024-03-25T19:27:33.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.384+0000 7f11167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:33.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.384+0000 7f11167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:33.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.384+0000 7f11167fc640 10 monclient(hunting): dump: 2024-03-25T19:27:33.384 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:33.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.384+0000 7f11167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:33.396 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.396+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1356833025 0 0) 0x7f11201dc6d0 con 0x7f11200a98f0 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 >> v1:172.21.15.60:6790/0 conn(0x7f11201430c0 legacy=0x7f1120138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 >> v1:172.21.15.60:6789/0 conn(0x7f112013f710 legacy=0x7f112006ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 1 monclient: found mon.b 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112006e680 con 0x7f11200a98f0 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 10 monclient: _finish_auth 0 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f11167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 5 monclient: authenticate success, global_id 4543 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 10 monclient: _renew_subs 2024-03-25T19:27:33.397 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:33.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112006f2e0 con 0x7f11200a98f0 2024-03-25T19:27:33.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:33.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 10 monclient: _renew_subs 2024-03-25T19:27:33.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:33.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1120139830 con 0x7f11200a98f0 2024-03-25T19:27:33.398 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.397+0000 7f1127c88640 1 librados: init done 2024-03-25T19:27:33.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.398+0000 7f1127c88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:33.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.398+0000 7f1127c88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:33.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.398+0000 7f1127c88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:33.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.398+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11201dbc30 con 0x7f11200a98f0 2024-03-25T19:27:33.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.667+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f111c004a90 con 0x7f11200a98f0 2024-03-25T19:27:33.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.667+0000 7f11167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.668+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f111c004da0 con 0x7f11200a98f0 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.668+0000 7f11167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.668+0000 7f11167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.668+0000 7f11167fc640 10 monclient: dump: 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:33.668 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:33.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.668+0000 7f1117fff640 4 set_mon_vals no callback set 2024-03-25T19:27:33.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.669+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f111c01ad70 con 0x7f11200a98f0 2024-03-25T19:27:33.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.669+0000 7f1125223640 10 monclient: get_auth_request con 0x7f110406ddf0 auth_method 0 2024-03-25T19:27:33.669 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.670+0000 7f11167fc640 10 client.4543.objecter ms_handle_connect 0x7f110406ddf0 2024-03-25T19:27:33.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.675+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f111c054980 con 0x7f11200a98f0 2024-03-25T19:27:33.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.675+0000 7f11167fc640 10 client.4543.objecter ms_dispatch 0x7f1120000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:33.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.675+0000 7f11167fc640 3 client.4543.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:33.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.675+0000 7f11167fc640 3 client.4543.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:33.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.675+0000 7f11167fc640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-25T19:27:33.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.676+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f111c0502c0 con 0x7f11200a98f0 2024-03-25T19:27:33.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.676+0000 7f11167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:33.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.676+0000 7f11167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:33.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.902+0000 7f1127c88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:33.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.902+0000 7f1127c88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:33.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.902+0000 7f1127c88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:33.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.902+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11200627b0 con 0x7f11200a98f0 2024-03-25T19:27:33.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.930+0000 7f11167fc640 1 -- 172.21.15.60:0/1526787715 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f111c0523c0 con 0x7f11200a98f0 2024-03-25T19:27:33.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.930+0000 7f11167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:33.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.930+0000 7f11167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:33.930 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:33.930 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:33.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 10 librados: watch_flush enter 2024-03-25T19:27:33.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 10 librados: watch_flush exit 2024-03-25T19:27:33.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-25T19:27:33.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 10 client.4543.objecter successfully canceled tick 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 >> v1:172.21.15.90:6800/1827638754 conn(0x7f110406ddf0 legacy=0x7f11040702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 10 monclient: shutdown 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 >> v1:172.21.15.90:6789/0 conn(0x7f11200a98f0 legacy=0x7f11201dbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 shutdown_connections 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 >> 172.21.15.60:0/1526787715 conn(0x7f11200b7ea0 msgr2=0x7f112013c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.932+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 shutdown_connections 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.933+0000 7f1127c88640 1 -- 172.21.15.60:0/1526787715 wait complete. 2024-03-25T19:27:33.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:33.933+0000 7f1127c88640 1 librados: shutdown 2024-03-25T19:27:33.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:40.330 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:41.041 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.341 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:41.053 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.382 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:07:41.093 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.383 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:41.095 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.404 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:41.115 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.408 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:41.062 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.414 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:41.068 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.444 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:07:41.155 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.454 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:07:41.108 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:27:40.752 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:40.875 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.876+0000 7f23574c1640 10 monclient: get_monmap_and_config 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.876+0000 7f23574c1640 10 monclient: build_initial_monmap 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.876+0000 7f23574c1640 10 monclient: monmap: 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:40.876979+0000 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:40.876979+0000 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:40.876 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:40.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.877+0000 7f23574c1640 1 Processor -- start 2024-03-25T19:27:40.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.877+0000 7f23574c1640 1 -- start start 2024-03-25T19:27:40.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.877+0000 7f23574c1640 10 monclient: init 2024-03-25T19:27:40.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.877+0000 7f23574c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:40.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.877+0000 7f23574c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:27:40.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 10 monclient(hunting): picked mon.noname-b con 0x7f23500b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:40.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 10 monclient(hunting): picked mon.noname-c con 0x7f23500b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:40.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 10 monclient(hunting): picked mon.noname-a con 0x7f235014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:40.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23500af940 con 0x7f235014be90 2024-03-25T19:27:40.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23500b0a70 con 0x7f23500b1730 2024-03-25T19:27:40.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23500b0d50 con 0x7f23500b4d60 2024-03-25T19:27:40.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:40.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.878+0000 7f23574c1640 10 monclient(hunting): authenticate will time out at 6787.976180s 2024-03-25T19:27:40.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.886+0000 7f2355a5e640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f235014be90 0x7f235014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33386/0 (socket says 172.21.15.60:33386) 2024-03-25T19:27:40.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.886+0000 7f2355a5e640 1 -- 172.21.15.60:0/1604349477 learned_addr learned my addr 172.21.15.60:0/1604349477 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:40.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.894+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964738117 0 0) 0x7f23500af940 con 0x7f235014be90 2024-03-25T19:27:40.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.894+0000 7f2347fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:40.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.894+0000 7f2347fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:40.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.894+0000 7f2347fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:40.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.894+0000 7f2347fff640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:27:40.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.894+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f233c003520 con 0x7f235014be90 2024-03-25T19:27:40.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 782826129 0 0) 0x7f233c003520 con 0x7f235014be90 2024-03-25T19:27:40.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 >> v1:172.21.15.60:6790/0 conn(0x7f23500b4d60 legacy=0x7f235013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:40.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 >> v1:172.21.15.90:6789/0 conn(0x7f23500b1730 legacy=0x7f23500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:40.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:40.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 1 monclient: found mon.noname-a 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23500b1030 con 0x7f235014be90 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2348001a10 con 0x7f235014be90 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f23574c1640 5 monclient: authenticate success, global_id 4554 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f23574c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f2347fff640 10 monclient: dump: 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:40.906 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:40.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.905+0000 7f23574c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:40.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.918+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2348001b70 con 0x7f235014be90 2024-03-25T19:27:40.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.918+0000 7f2347fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:40.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.918+0000 7f2347fff640 1 -- 172.21.15.60:0/1604349477 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2348001f80 con 0x7f235014be90 2024-03-25T19:27:40.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.918+0000 7f2347fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:40.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.918+0000 7f2347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:40.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.918+0000 7f2347fff640 10 monclient: dump: 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.919+0000 7f23574c1640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:40.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.919+0000 7f23574c1640 4 set_mon_vals no callback set 2024-03-25T19:27:40.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.919+0000 7f23574c1640 10 monclient: shutdown 2024-03-25T19:27:40.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.919+0000 7f23574c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.919+0000 7f23574c1640 1 -- 172.21.15.60:0/1604349477 >> v1:172.21.15.60:6789/0 conn(0x7f235014be90 legacy=0x7f235014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.919+0000 7f23574c1640 1 -- 172.21.15.60:0/1604349477 shutdown_connections 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.919+0000 7f23574c1640 1 -- 172.21.15.60:0/1604349477 >> 172.21.15.60:0/1604349477 conn(0x7f23500617a0 msgr2=0x7f2350061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.920+0000 7f23574c1640 1 -- 172.21.15.60:0/1604349477 shutdown_connections 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.920+0000 7f23574c1640 1 -- 172.21.15.60:0/1604349477 wait complete. 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.920+0000 7f23574c1640 10 monclient: build_initial_monmap 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.920+0000 7f23574c1640 10 monclient: monmap: 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:40.921 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 1 librados: starting msgr at 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 1 librados: starting objecter 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 1 Processor -- start 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 1 -- start start 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 1 librados: setting wanted keys 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 1 librados: calling monclient init 2024-03-25T19:27:40.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 10 monclient: init 2024-03-25T19:27:40.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:40.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:40.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 10 monclient(hunting): picked mon.noname-a con 0x7f23500b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:40.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 10 monclient(hunting): picked mon.noname-b con 0x7f23500b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:40.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.921+0000 7f23574c1640 10 monclient(hunting): picked mon.noname-c con 0x7f235014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:40.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.922+0000 7f23574c1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2350140380 con 0x7f23500b1730 2024-03-25T19:27:40.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.922+0000 7f23574c1640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2350140660 con 0x7f23500b4d60 2024-03-25T19:27:40.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.922+0000 7f23574c1640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2350141e30 con 0x7f235014be90 2024-03-25T19:27:40.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.922+0000 7f23574c1640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:40.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.922+0000 7f23574c1640 10 monclient(hunting): authenticate will time out at 6788.019972s 2024-03-25T19:27:40.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.930+0000 7f2354a5c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f23500b4d60 0x7f2350142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:44944/0 (socket says 172.21.15.60:44944) 2024-03-25T19:27:40.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.930+0000 7f2354a5c640 1 -- 172.21.15.60:0/1940344989 learned_addr learned my addr 172.21.15.60:0/1940344989 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:40.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.931+0000 7f2345ffb640 10 client.?.objecter ms_handle_connect 0x7f23500b4d60 2024-03-25T19:27:40.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.931+0000 7f2345ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:40.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.938+0000 7f2345ffb640 10 client.?.objecter ms_handle_connect 0x7f23500b1730 2024-03-25T19:27:40.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.938+0000 7f2345ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:40.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.943+0000 7f2345ffb640 10 client.?.objecter ms_handle_connect 0x7f235014be90 2024-03-25T19:27:40.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.943+0000 7f2345ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.954+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074523803 0 0) 0x7f2350140660 con 0x7f23500b4d60 2024-03-25T19:27:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.954+0000 7f2345ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.954+0000 7f2345ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.954+0000 7f2345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.954+0000 7f2345ffb640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:27:40.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.954+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2334004eb0 con 0x7f23500b4d60 2024-03-25T19:27:40.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.965+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393724544 0 0) 0x7f2350140380 con 0x7f23500b1730 2024-03-25T19:27:40.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.965+0000 7f2345ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:40.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.965+0000 7f2345ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:40.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.965+0000 7f2345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:40.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.965+0000 7f2345ffb640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:27:40.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.965+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2350140660 con 0x7f23500b1730 2024-03-25T19:27:40.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.972+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053606096 0 0) 0x7f2350141e30 con 0x7f235014be90 2024-03-25T19:27:40.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.972+0000 7f2345ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:40.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.972+0000 7f2345ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:40.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.972+0000 7f2345ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:40.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.972+0000 7f2345ffb640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:27:40.972 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.972+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2350140380 con 0x7f235014be90 2024-03-25T19:27:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.984+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1670226313 0 0) 0x7f2334004eb0 con 0x7f23500b4d60 2024-03-25T19:27:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.984+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2350141e30 con 0x7f23500b4d60 2024-03-25T19:27:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.984+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f23400019d0 con 0x7f23500b4d60 2024-03-25T19:27:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.984+0000 7f2345ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:40.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.984+0000 7f2345ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.984+0000 7f2345ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:40.984 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.984+0000 7f2345ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.985+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2208320637 0 0) 0x7f2350140380 con 0x7f235014be90 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.985+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2334004eb0 con 0x7f235014be90 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.985+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2348001e80 con 0x7f235014be90 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.985+0000 7f2345ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.985+0000 7f2345ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.985+0000 7f2345ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:40.985 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:40.986 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:40.986 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:40.986 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.996+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 726407874 0 0) 0x7f2350140660 con 0x7f23500b1730 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.996+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2350140380 con 0x7f23500b1730 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.996+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f234c0018b0 con 0x7f23500b1730 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.996+0000 7f2345ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.996+0000 7f2345ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:40.996+0000 7f2345ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:40.996 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:40.997 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:40.997 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 700377720 0 0) 0x7f2350141e30 con 0x7f23500b4d60 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 >> v1:172.21.15.60:6790/0 conn(0x7f235014be90 legacy=0x7f23501e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 >> v1:172.21.15.60:6789/0 conn(0x7f23500b1730 legacy=0x7f2350138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 1 monclient: found mon.b 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2350142110 con 0x7f23500b4d60 2024-03-25T19:27:41.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f2345ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 5 monclient: authenticate success, global_id 4549 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 10 monclient: _renew_subs 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2350140e60 con 0x7f23500b4d60 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 10 monclient: _renew_subs 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2350142c40 con 0x7f23500b4d60 2024-03-25T19:27:41.005 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.004+0000 7f23574c1640 1 librados: init done 2024-03-25T19:27:41.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.005+0000 7f23574c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:41.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.005+0000 7f23574c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:41.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.005+0000 7f23574c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:41.006 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.005+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23500627b0 con 0x7f23500b4d60 2024-03-25T19:27:41.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2340003610 con 0x7f23500b4d60 2024-03-25T19:27:41.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f2345ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:41.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2340003a20 con 0x7f23500b4d60 2024-03-25T19:27:41.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f2345ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:41.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f2345ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:41.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f2345ffb640 10 monclient: dump: 2024-03-25T19:27:41.294 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:41.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f23477fe640 4 set_mon_vals no callback set 2024-03-25T19:27:41.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.294+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2340003c40 con 0x7f23500b4d60 2024-03-25T19:27:41.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.295+0000 7f235525d640 10 monclient: get_auth_request con 0x7f233406e0f0 auth_method 0 2024-03-25T19:27:41.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.295+0000 7f2345ffb640 10 client.4549.objecter ms_handle_connect 0x7f233406e0f0 2024-03-25T19:27:41.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f23400547d0 con 0x7f23500b4d60 2024-03-25T19:27:41.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 10 client.4549.objecter ms_dispatch 0x7f2350000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:41.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:41.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:41.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T19:27:41.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23400502e0 con 0x7f23500b4d60 2024-03-25T19:27:41.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:41.299 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.299+0000 7f2345ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:41.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.579+0000 7f23574c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:41.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.579+0000 7f23574c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:41.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.579+0000 7f23574c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:41.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.579+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23500b5cf0 con 0x7f23500b4d60 2024-03-25T19:27:41.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.688+0000 7f2345ffb640 1 -- 172.21.15.60:0/1940344989 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f2340052210 con 0x7f23500b4d60 2024-03-25T19:27:41.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.688+0000 7f2345ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:41.688 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.688+0000 7f2345ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:41.688 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:41.688 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.690+0000 7f23574c1640 10 librados: watch_flush enter 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 10 librados: watch_flush exit 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 10 client.4549.objecter successfully canceled tick 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 >> v1:172.21.15.90:6800/1827638754 conn(0x7f233406e0f0 legacy=0x7f23340705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 10 monclient: shutdown 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 >> v1:172.21.15.90:6789/0 conn(0x7f23500b4d60 legacy=0x7f2350142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:41.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 shutdown_connections 2024-03-25T19:27:41.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 >> 172.21.15.60:0/1940344989 conn(0x7f23500617a0 msgr2=0x7f23500b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:41.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 shutdown_connections 2024-03-25T19:27:41.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 1 -- 172.21.15.60:0/1940344989 wait complete. 2024-03-25T19:27:41.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:41.691+0000 7f23574c1640 1 librados: shutdown 2024-03-25T19:27:41.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:48.508 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.609+0000 7f28a25fb640 10 monclient: get_monmap_and_config 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.609+0000 7f28a25fb640 10 monclient: build_initial_monmap 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.609+0000 7f28a25fb640 10 monclient: monmap: 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:48.610551+0000 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:48.610551+0000 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:48.609 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:48.610 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:48.610 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:48.610 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:48.610 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.610+0000 7f28a25fb640 1 Processor -- start 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.610+0000 7f28a25fb640 1 -- start start 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.610+0000 7f28a25fb640 10 monclient: init 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 10 monclient(hunting): picked mon.noname-a con 0x7f289c0a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 10 monclient(hunting): picked mon.noname-c con 0x7f289c13f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 10 monclient(hunting): picked mon.noname-b con 0x7f289c1430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c142520 con 0x7f289c0a98f0 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c0b28a0 con 0x7f289c13f710 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c0b2a30 con 0x7f289c1430c0 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:48.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.611+0000 7f28a25fb640 10 monclient(hunting): authenticate will time out at 6795.709162s 2024-03-25T19:27:48.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.620+0000 7f289bfff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f289c0a98f0 0x7f289c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34324/0 (socket says 172.21.15.60:34324) 2024-03-25T19:27:48.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.620+0000 7f289bfff640 1 -- 172.21.15.60:0/1535528597 learned_addr learned my addr 172.21.15.60:0/1535528597 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:48.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.628+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138548635 0 0) 0x7f289c142520 con 0x7f289c0a98f0 2024-03-25T19:27:48.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.628+0000 7f289affd640 10 monclient(hunting): _negotiate 2024-03-25T19:27:48.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.628+0000 7f289affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:48.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.628+0000 7f289affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:48.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.628+0000 7f289affd640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:27:48.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.628+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2884003520 con 0x7f289c0a98f0 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4039272270 0 0) 0x7f2884003520 con 0x7f289c0a98f0 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 >> v1:172.21.15.60:6790/0 conn(0x7f289c1430c0 legacy=0x7f289c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 >> v1:172.21.15.90:6789/0 conn(0x7f289c13f710 legacy=0x7f289c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 10 monclient: _finish_hunting 0 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 1 monclient: found mon.noname-a 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c0b44c0 con 0x7f289c0a98f0 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 10 monclient: _finish_auth 0 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 10 monclient: _check_auth_tickets 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:48.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f288c001690 con 0x7f289c0a98f0 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f289affd640 10 monclient: dump: 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f28a25fb640 5 monclient: authenticate success, global_id 4560 2024-03-25T19:27:48.643 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.639+0000 7f28a25fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f288c001860 con 0x7f289c0a98f0 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f289affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f289affd640 1 -- 172.21.15.60:0/1535528597 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f288c001cb0 con 0x7f289c0a98f0 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f289affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f289affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f289affd640 10 monclient: dump: 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:48.649 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:48.650 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:48.650 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:48.650 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:48.650 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:48.650 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f28a25fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:48.650 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.649+0000 7f28a25fb640 4 set_mon_vals no callback set 2024-03-25T19:27:48.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.650+0000 7f28a25fb640 10 monclient: shutdown 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.650+0000 7f28a25fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.650+0000 7f28a25fb640 1 -- 172.21.15.60:0/1535528597 >> v1:172.21.15.60:6789/0 conn(0x7f289c0a98f0 legacy=0x7f289c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.650+0000 7f28a25fb640 1 -- 172.21.15.60:0/1535528597 shutdown_connections 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.650+0000 7f28a25fb640 1 -- 172.21.15.60:0/1535528597 >> 172.21.15.60:0/1535528597 conn(0x7f289c0b7ea0 msgr2=0x7f289c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.650+0000 7f28a25fb640 1 -- 172.21.15.60:0/1535528597 shutdown_connections 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.650+0000 7f28a25fb640 1 -- 172.21.15.60:0/1535528597 wait complete. 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.651+0000 7f28a25fb640 10 monclient: build_initial_monmap 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.651+0000 7f28a25fb640 10 monclient: monmap: 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:48.652 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.652+0000 7f28a25fb640 1 librados: starting msgr at 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.652+0000 7f28a25fb640 1 librados: starting objecter 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.652+0000 7f28a25fb640 1 Processor -- start 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.652+0000 7f28a25fb640 1 -- start start 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.652+0000 7f28a25fb640 1 librados: setting wanted keys 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.652+0000 7f28a25fb640 1 librados: calling monclient init 2024-03-25T19:27:48.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.652+0000 7f28a25fb640 10 monclient: init 2024-03-25T19:27:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 10 monclient(hunting): picked mon.noname-c con 0x7f289c0a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:48.654 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 10 monclient(hunting): picked mon.noname-a con 0x7f289c13f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 10 monclient(hunting): picked mon.noname-b con 0x7f289c1430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c0abb10 con 0x7f289c13f710 2024-03-25T19:27:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c1d7850 con 0x7f289c1430c0 2024-03-25T19:27:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289c1d7500 con 0x7f289c0a98f0 2024-03-25T19:27:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:48.655 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.653+0000 7f28a25fb640 10 monclient(hunting): authenticate will time out at 6795.751257s 2024-03-25T19:27:48.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.667+0000 7f289b7fe640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f289c13f710 0x7f289c1d7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:34344/0 (socket says 172.21.15.60:34344) 2024-03-25T19:27:48.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.667+0000 7f289b7fe640 1 -- 172.21.15.60:0/284182504 learned_addr learned my addr 172.21.15.60:0/284182504 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:48.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.671+0000 7f2898ff9640 10 client.?.objecter ms_handle_connect 0x7f289c1430c0 2024-03-25T19:27:48.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.671+0000 7f2898ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:48.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.673+0000 7f2898ff9640 10 client.?.objecter ms_handle_connect 0x7f289c13f710 2024-03-25T19:27:48.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.673+0000 7f2898ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:48.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.677+0000 7f2898ff9640 10 client.?.objecter ms_handle_connect 0x7f289c0a98f0 2024-03-25T19:27:48.676 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.677+0000 7f2898ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:48.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.680+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652140254 0 0) 0x7f289c1d7850 con 0x7f289c1430c0 2024-03-25T19:27:48.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.680+0000 7f2898ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:48.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.680+0000 7f2898ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:48.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.680+0000 7f2898ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:48.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.680+0000 7f2898ff9640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:27:48.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.680+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2888004eb0 con 0x7f289c1430c0 2024-03-25T19:27:48.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.681+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788399751 0 0) 0x7f289c0abb10 con 0x7f289c13f710 2024-03-25T19:27:48.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.681+0000 7f2898ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:27:48.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.681+0000 7f2898ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:48.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.681+0000 7f2898ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:48.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.681+0000 7f2898ff9640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:27:48.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.681+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f289c1d7850 con 0x7f289c13f710 2024-03-25T19:27:48.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.692+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3155875266 0 0) 0x7f289c1d7850 con 0x7f289c13f710 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.692+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f289c0abb10 con 0x7f289c13f710 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.692+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f28900018e0 con 0x7f289c13f710 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.692+0000 7f2898ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.692+0000 7f2898ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.692+0000 7f2898ff9640 10 monclient(hunting): dump: 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:48.692 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:48.693 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:48.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.692+0000 7f2898ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4084488578 0 0) 0x7f289c0abb10 con 0x7f289c13f710 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 >> v1:172.21.15.60:6790/0 conn(0x7f289c0a98f0 legacy=0x7f289c1384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 >> v1:172.21.15.90:6789/0 conn(0x7f289c1430c0 legacy=0x7f289c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 1 monclient: found mon.a 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c137750 con 0x7f289c13f710 2024-03-25T19:27:48.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 10 monclient: _finish_auth 0 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f2898ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 5 monclient: authenticate success, global_id 4563 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 10 monclient: _renew_subs 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289c137a00 con 0x7f289c13f710 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 10 monclient: _renew_subs 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f289c134800 con 0x7f289c13f710 2024-03-25T19:27:48.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.699+0000 7f28a25fb640 1 librados: init done 2024-03-25T19:27:48.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.700+0000 7f28a25fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:48.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.700+0000 7f28a25fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:48.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.700+0000 7f28a25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:48.701 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:48.700+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f289c1d7690 con 0x7f289c13f710 2024-03-25T19:27:49.073 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.073+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28900035f0 con 0x7f289c13f710 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.074+0000 7f2898ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.074+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2890003940 con 0x7f289c13f710 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.074+0000 7f2898ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.074+0000 7f2898ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.074+0000 7f2898ff9640 10 monclient: dump: 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:49.074 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:49.075 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:49.075 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:49.075 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:49.075 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:49.075 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:49.075 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.074+0000 7f289a7fc640 4 set_mon_vals no callback set 2024-03-25T19:27:49.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.074+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2890003b60 con 0x7f289c13f710 2024-03-25T19:27:49.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.075+0000 7f289bfff640 10 monclient: get_auth_request con 0x7f288806dd60 auth_method 0 2024-03-25T19:27:49.076 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.075+0000 7f2898ff9640 10 client.4563.objecter ms_handle_connect 0x7f288806dd60 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f2890054720 con 0x7f289c13f710 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 10 client.4563.objecter ms_dispatch 0x7f289c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2890050230 con 0x7f289c13f710 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:49.081 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.081+0000 7f2898ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:49.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.334+0000 7f28a25fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:49.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.334+0000 7f28a25fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:49.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.334+0000 7f28a25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:27:49.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.334+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f289c1d79e0 con 0x7f289c13f710 2024-03-25T19:27:49.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.363+0000 7f2898ff9640 1 -- 172.21.15.60:0/284182504 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f2890052160 con 0x7f289c13f710 2024-03-25T19:27:49.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.363+0000 7f2898ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:49.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.363+0000 7f2898ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:49.363 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:49.363 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.365+0000 7f28a25fb640 10 librados: watch_flush enter 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 10 librados: watch_flush exit 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 10 client.4563.objecter successfully canceled tick 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 >> v1:172.21.15.90:6800/1827638754 conn(0x7f288806dd60 legacy=0x7f2888070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 10 monclient: shutdown 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 >> v1:172.21.15.60:6789/0 conn(0x7f289c13f710 legacy=0x7f289c1d7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 shutdown_connections 2024-03-25T19:27:49.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 >> 172.21.15.60:0/284182504 conn(0x7f289c0b7ea0 msgr2=0x7f289c13ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:49.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 shutdown_connections 2024-03-25T19:27:49.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 1 -- 172.21.15.60:0/284182504 wait complete. 2024-03-25T19:27:49.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:49.366+0000 7f28a25fb640 1 librados: shutdown 2024-03-25T19:27:49.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:27:56.185 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.302+0000 7ff9b3515640 10 monclient: get_monmap_and_config 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.302+0000 7ff9b3515640 10 monclient: build_initial_monmap 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.302+0000 7ff9b3515640 10 monclient: monmap: 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:27:56.304089+0000 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:27:56.304089+0000 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:27:56.303 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:27:56.304 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.303+0000 7ff9b3515640 1 Processor -- start 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.303+0000 7ff9b3515640 1 -- start start 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.303+0000 7ff9b3515640 10 monclient: init 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 10 monclient(hunting): picked mon.noname-c con 0x7ff9ac0a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 10 monclient(hunting): picked mon.noname-b con 0x7ff9ac13f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 10 monclient(hunting): picked mon.noname-a con 0x7ff9ac1430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:56.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac142520 con 0x7ff9ac1430c0 2024-03-25T19:27:56.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac0b28a0 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac0b2a30 con 0x7ff9ac13f710 2024-03-25T19:27:56.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:56.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.304+0000 7ff9b3515640 10 monclient(hunting): authenticate will time out at 6803.403218s 2024-03-25T19:27:56.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.310+0000 7ff9b1ab2640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7ff9ac1430c0 0x7ff9ac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:33556/0 (socket says 172.21.15.60:33556) 2024-03-25T19:27:56.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.310+0000 7ff9b1ab2640 1 -- 172.21.15.60:0/1320620449 learned_addr learned my addr 172.21.15.60:0/1320620449 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:56.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.337+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608636378 0 0) 0x7ff9ac0b28a0 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.337+0000 7ff9a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:56.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.337+0000 7ff9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:56.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.337+0000 7ff9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:56.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.337+0000 7ff9a3fff640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:27:56.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.337+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff998003520 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.343+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981992783 0 0) 0x7ff9ac142520 con 0x7ff9ac1430c0 2024-03-25T19:27:56.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.343+0000 7ff9a3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:27:56.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.343+0000 7ff9a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:56.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.343+0000 7ff9a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:56.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.343+0000 7ff9a3fff640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:27:56.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.343+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9ac0b28a0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3646329623 0 0) 0x7ff998003520 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 >> v1:172.21.15.60:6790/0 conn(0x7ff9ac13f710 legacy=0x7ff9ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:56.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 >> v1:172.21.15.60:6789/0 conn(0x7ff9ac1430c0 legacy=0x7ff9ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:56.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 1 monclient: found mon.noname-c 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0b44c0 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 10 monclient: _finish_auth 0 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff9a8001690 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9a3fff640 10 monclient: dump: 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.347 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9b3515640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:27:56.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.345+0000 7ff9b3515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9a80017f0 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9a3fff640 1 -- 172.21.15.60:0/1320620449 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff9a8001c00 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9b3515640 10 monclient: get_monmap_and_config success 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9b3515640 4 set_mon_vals no callback set 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.359+0000 7ff9a3fff640 10 monclient: dump: 2024-03-25T19:27:56.360 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:56.361 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:56.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.360+0000 7ff9b3515640 10 monclient: shutdown 2024-03-25T19:27:56.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.360+0000 7ff9b3515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:56.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.360+0000 7ff9b3515640 1 -- 172.21.15.60:0/1320620449 >> v1:172.21.15.90:6789/0 conn(0x7ff9ac0a98f0 legacy=0x7ff9ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:56.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.360+0000 7ff9b3515640 1 -- 172.21.15.60:0/1320620449 shutdown_connections 2024-03-25T19:27:56.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.360+0000 7ff9b3515640 1 -- 172.21.15.60:0/1320620449 >> 172.21.15.60:0/1320620449 conn(0x7ff9ac0b7ea0 msgr2=0x7ff9ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:56.363 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.361+0000 7ff9b3515640 1 -- 172.21.15.60:0/1320620449 shutdown_connections 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.361+0000 7ff9b3515640 1 -- 172.21.15.60:0/1320620449 wait complete. 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.361+0000 7ff9b3515640 10 monclient: build_initial_monmap 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.361+0000 7ff9b3515640 10 monclient: monmap: 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:56.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 librados: starting msgr at 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 librados: starting objecter 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 Processor -- start 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 -- start start 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 librados: setting wanted keys 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 librados: calling monclient init 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient: init 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient: _reopen_session rank -1 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient(hunting): picked mon.noname-c con 0x7ff9ac0a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient(hunting): picked mon.noname-a con 0x7ff9ac13f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient(hunting): picked mon.noname-b con 0x7ff9ac1430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac1daba0 con 0x7ff9ac13f710 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac0abac0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac1db740 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient(hunting): _renew_subs 2024-03-25T19:27:56.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.362+0000 7ff9b3515640 10 monclient(hunting): authenticate will time out at 6803.461872s 2024-03-25T19:27:56.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.370+0000 7ff9b12b1640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7ff9ac0a98f0 0x7ff9ac1db030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:51682/0 (socket says 172.21.15.60:51682) 2024-03-25T19:27:56.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.370+0000 7ff9b12b1640 1 -- 172.21.15.60:0/2856931748 learned_addr learned my addr 172.21.15.60:0/2856931748 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:27:56.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.371+0000 7ff9a1ffb640 10 client.?.objecter ms_handle_connect 0x7ff9ac0a98f0 2024-03-25T19:27:56.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.371+0000 7ff9a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:56.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.380+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745304452 0 0) 0x7ff9ac1db740 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.380+0000 7ff9a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:56.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.380+0000 7ff9a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:56.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.380+0000 7ff9a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:56.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.380+0000 7ff9a1ffb640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:27:56.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.380+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff990004eb0 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.383+0000 7ff9a1ffb640 10 client.?.objecter ms_handle_connect 0x7ff9ac1430c0 2024-03-25T19:27:56.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.383+0000 7ff9a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:56.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.388+0000 7ff9a1ffb640 10 client.?.objecter ms_handle_connect 0x7ff9ac13f710 2024-03-25T19:27:56.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.388+0000 7ff9a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:27:56.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.398+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571230491 0 0) 0x7ff9ac0abac0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.398+0000 7ff9a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:56.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.398+0000 7ff9a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:56.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.398+0000 7ff9a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:56.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.398+0000 7ff9a1ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:27:56.399 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.398+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9ac1db740 con 0x7ff9ac1430c0 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.399+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228656265 0 0) 0x7ff990004eb0 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.399+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9ac0abac0 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.399+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff9a8001690 con 0x7ff9ac0a98f0 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.399+0000 7ff9a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.399+0000 7ff9a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.399+0000 7ff9a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:56.400 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:56.401 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.399+0000 7ff9a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:27:56.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.408+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867177575 0 0) 0x7ff9ac1daba0 con 0x7ff9ac13f710 2024-03-25T19:27:56.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.408+0000 7ff9a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:27:56.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.408+0000 7ff9a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:27:56.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.408+0000 7ff9a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:27:56.408 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.408+0000 7ff9a1ffb640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:27:56.409 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.408+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff990004eb0 con 0x7ff9ac13f710 2024-03-25T19:27:56.412 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.412+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 792978911 0 0) 0x7ff9ac1db740 con 0x7ff9ac1430c0 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.412+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9ac1daba0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.412+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff9a4001b90 con 0x7ff9ac1430c0 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.412+0000 7ff9a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.412+0000 7ff9a1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.412+0000 7ff9a1ffb640 10 monclient(hunting): dump: 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:56.413 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:56.414 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:56.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1568277964 0 0) 0x7ff9ac1daba0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 >> v1:172.21.15.60:6790/0 conn(0x7ff9ac0a98f0 legacy=0x7ff9ac1db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:56.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 >> v1:172.21.15.60:6789/0 conn(0x7ff9ac13f710 legacy=0x7ff9ac138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:56.429 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 1 monclient: found mon.b 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac1db8d0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 5 monclient: authenticate success, global_id 4558 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 10 monclient: _renew_subs 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac138c10 con 0x7ff9ac1430c0 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:27:56.430 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 10 monclient: _renew_subs 2024-03-25T19:27:56.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:56.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac1350e0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.428+0000 7ff9b3515640 1 librados: init done 2024-03-25T19:27:56.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.429+0000 7ff9b3515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:56.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.429+0000 7ff9b3515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:56.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.429+0000 7ff9b3515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:56.431 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.429+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac0627b0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.704+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9a4002040 con 0x7ff9ac1430c0 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.704+0000 7ff9a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.704+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7ff9a4004d00 con 0x7ff9ac1430c0 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.704+0000 7ff9a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.704+0000 7ff9a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.704+0000 7ff9a1ffb640 10 monclient: dump: 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:27:56.705 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:27:56.706 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.704+0000 7ff9a37fe640 4 set_mon_vals no callback set 2024-03-25T19:27:56.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.705+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7ff9a401ad10 con 0x7ff9ac1430c0 2024-03-25T19:27:56.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.705+0000 7ff9b12b1640 10 monclient: get_auth_request con 0x7ff99006df50 auth_method 0 2024-03-25T19:27:56.707 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.706+0000 7ff9a1ffb640 10 client.4558.objecter ms_handle_connect 0x7ff99006df50 2024-03-25T19:27:56.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.709+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7ff9a4054a20 con 0x7ff9ac1430c0 2024-03-25T19:27:56.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.709+0000 7ff9a1ffb640 10 client.4558.objecter ms_dispatch 0x7ff9ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:27:56.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.709+0000 7ff9a1ffb640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:27:56.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.709+0000 7ff9a1ffb640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:27:56.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.710+0000 7ff9a1ffb640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T19:27:56.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.710+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9a4050360 con 0x7ff9ac1430c0 2024-03-25T19:27:56.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.710+0000 7ff9a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:27:56.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.710+0000 7ff9a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:27:56.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.934+0000 7ff9b3515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:56.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.934+0000 7ff9b3515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:56.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.934+0000 7ff9b3515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:27:56.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.934+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9ac074cc0 con 0x7ff9ac1430c0 2024-03-25T19:27:56.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.970+0000 7ff9a1ffb640 1 -- 172.21.15.60:0/2856931748 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7ff9a4052460 con 0x7ff9ac1430c0 2024-03-25T19:27:56.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.970+0000 7ff9a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:27:56.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.970+0000 7ff9a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:27:56.971 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:27:56.971 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 10 librados: watch_flush enter 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 10 librados: watch_flush exit 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 10 client.4558.objecter successfully canceled tick 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 >> v1:172.21.15.90:6800/1827638754 conn(0x7ff99006df50 legacy=0x7ff990070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 10 monclient: shutdown 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 >> v1:172.21.15.90:6789/0 conn(0x7ff9ac1430c0 legacy=0x7ff9ac1e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:27:56.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 shutdown_connections 2024-03-25T19:27:56.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 >> 172.21.15.60:0/2856931748 conn(0x7ff9ac0b7ea0 msgr2=0x7ff9ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:27:56.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 shutdown_connections 2024-03-25T19:27:56.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 1 -- 172.21.15.60:0/2856931748 wait complete. 2024-03-25T19:27:56.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:27:56.973+0000 7ff9b3515640 1 librados: shutdown 2024-03-25T19:27:56.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:00.350 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:01.061 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.403 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:01.114 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.436 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:01.147 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.450 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:01.161 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.461 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:01.172 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.465 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:01.119 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.469 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:01.123 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.503 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:01.214 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:00.509 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:01.163 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:03.791 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:03.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.905+0000 7f7379f04640 10 monclient: get_monmap_and_config 2024-03-25T19:28:03.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.905+0000 7f7379f04640 10 monclient: build_initial_monmap 2024-03-25T19:28:03.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.905+0000 7f7379f04640 10 monclient: monmap: 2024-03-25T19:28:03.905 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:03.906758+0000 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:03.906758+0000 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:03.906 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:03.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.905+0000 7f7379f04640 1 Processor -- start 2024-03-25T19:28:03.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.905+0000 7f7379f04640 1 -- start start 2024-03-25T19:28:03.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.905+0000 7f7379f04640 10 monclient: init 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 10 monclient(hunting): picked mon.noname-b con 0x7f73740705b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 10 monclient(hunting): picked mon.noname-a con 0x7f73741402d0 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 10 monclient(hunting): picked mon.noname-c con 0x7f7374149ae0 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73740718c0 con 0x7f73741402d0 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73740714c0 con 0x7f7374149ae0 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7374072b50 con 0x7f73740705b0 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:03.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.906+0000 7f7379f04640 10 monclient(hunting): authenticate will time out at 6811.005566s 2024-03-25T19:28:03.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.912+0000 7f7373fff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f7374149ae0 0x7f737414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:41338/0 (socket says 172.21.15.60:41338) 2024-03-25T19:28:03.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.912+0000 7f7373fff640 1 -- 172.21.15.60:0/3131884440 learned_addr learned my addr 172.21.15.60:0/3131884440 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:03.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.929+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465106739 0 0) 0x7f73740714c0 con 0x7f7374149ae0 2024-03-25T19:28:03.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.929+0000 7f73727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:03.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.929+0000 7f73727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:03.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.929+0000 7f73727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:03.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.929+0000 7f73727fc640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:28:03.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.929+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f735c003520 con 0x7f7374149ae0 2024-03-25T19:28:03.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.935+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450054071 0 0) 0x7f7374072b50 con 0x7f73740705b0 2024-03-25T19:28:03.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.935+0000 7f73727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:28:03.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.935+0000 7f73727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:03.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.935+0000 7f73727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:03.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.935+0000 7f73727fc640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:28:03.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.935+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73740714c0 con 0x7f73740705b0 2024-03-25T19:28:03.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.952+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1645320734 0 0) 0x7f735c003520 con 0x7f7374149ae0 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 >> v1:172.21.15.60:6790/0 conn(0x7f73740705b0 legacy=0x7f737413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 >> v1:172.21.15.60:6789/0 conn(0x7f73741402d0 legacy=0x7f7374140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 1 monclient: found mon.noname-c 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737413dd10 con 0x7f7374149ae0 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 10 monclient: _finish_auth 0 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f736c001a10 con 0x7f7374149ae0 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:03.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f73727fc640 10 monclient: dump: 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f7379f04640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:28:03.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.953+0000 7f7379f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:03.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f736c001b70 con 0x7f7374149ae0 2024-03-25T19:28:03.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f73727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:03.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f73727fc640 1 -- 172.21.15.60:0/3131884440 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f736c001f80 con 0x7f7374149ae0 2024-03-25T19:28:03.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f7379f04640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:03.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f7379f04640 4 set_mon_vals no callback set 2024-03-25T19:28:03.982 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f73727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:03.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f73727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:03.983 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.981+0000 7f73727fc640 10 monclient: dump: 2024-03-25T19:28:03.985 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:03.985 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:03.986 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.982+0000 7f7379f04640 10 monclient: shutdown 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.982+0000 7f7379f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.982+0000 7f7379f04640 1 -- 172.21.15.60:0/3131884440 >> v1:172.21.15.90:6789/0 conn(0x7f7374149ae0 legacy=0x7f737414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.982+0000 7f7379f04640 1 -- 172.21.15.60:0/3131884440 shutdown_connections 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.982+0000 7f7379f04640 1 -- 172.21.15.60:0/3131884440 >> 172.21.15.60:0/3131884440 conn(0x7f73740b5d10 msgr2=0x7f73740b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.982+0000 7f7379f04640 1 -- 172.21.15.60:0/3131884440 shutdown_connections 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.982+0000 7f7379f04640 1 -- 172.21.15.60:0/3131884440 wait complete. 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.983+0000 7f7379f04640 10 monclient: build_initial_monmap 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.983+0000 7f7379f04640 10 monclient: monmap: 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:03.988 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.984+0000 7f7379f04640 1 librados: starting msgr at 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.984+0000 7f7379f04640 1 librados: starting objecter 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.984+0000 7f7379f04640 1 Processor -- start 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.984+0000 7f7379f04640 1 -- start start 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.984+0000 7f7379f04640 1 librados: setting wanted keys 2024-03-25T19:28:03.989 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.984+0000 7f7379f04640 1 librados: calling monclient init 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.984+0000 7f7379f04640 10 monclient: init 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 10 monclient(hunting): picked mon.noname-c con 0x7f73740705b0 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 10 monclient(hunting): picked mon.noname-b con 0x7f73741402d0 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 10 monclient(hunting): picked mon.noname-a con 0x7f7374149ae0 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7374143890 con 0x7f7374149ae0 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7374144b70 con 0x7f73741402d0 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7374143370 con 0x7f73740705b0 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:03.990 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.985+0000 7f7379f04640 10 monclient(hunting): authenticate will time out at 6811.084599s 2024-03-25T19:28:03.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.996+0000 7f7372ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f73741402d0 0x7f7374141290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:41386/0 (socket says 172.21.15.60:41386) 2024-03-25T19:28:03.997 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.996+0000 7f7372ffd640 1 -- 172.21.15.60:0/2355779356 learned_addr learned my addr 172.21.15.60:0/2355779356 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:04.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.999+0000 7f735bfff640 10 client.?.objecter ms_handle_connect 0x7f73741402d0 2024-03-25T19:28:04.000 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:03.999+0000 7f735bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:04.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.012+0000 7f735bfff640 10 client.?.objecter ms_handle_connect 0x7f7374149ae0 2024-03-25T19:28:04.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.012+0000 7f735bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:04.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.012+0000 7f735bfff640 10 client.?.objecter ms_handle_connect 0x7f73740705b0 2024-03-25T19:28:04.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.012+0000 7f735bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:04.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.013+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492391278 0 0) 0x7f7374144b70 con 0x7f73741402d0 2024-03-25T19:28:04.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.013+0000 7f735bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:04.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.013+0000 7f735bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:04.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.013+0000 7f735bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:04.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.013+0000 7f735bfff640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:28:04.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.013+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7360004eb0 con 0x7f73741402d0 2024-03-25T19:28:04.027 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.027+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64348089 0 0) 0x7f7374143370 con 0x7f73740705b0 2024-03-25T19:28:04.027 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.027+0000 7f735bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:04.027 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.027+0000 7f735bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:04.027 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.027+0000 7f735bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:04.027 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.027+0000 7f735bfff640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:28:04.027 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.027+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7374144b70 con 0x7f73740705b0 2024-03-25T19:28:04.036 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.035+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2556390348 0 0) 0x7f7360004eb0 con 0x7f73741402d0 2024-03-25T19:28:04.036 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.035+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7374143370 con 0x7f73741402d0 2024-03-25T19:28:04.036 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7368001a30 con 0x7f73741402d0 2024-03-25T19:28:04.036 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): dump: 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628750218 0 0) 0x7f7374143890 con 0x7f7374149ae0 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:04.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:04.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:04.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:28:04.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.036+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7360004eb0 con 0x7f7374149ae0 2024-03-25T19:28:04.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.037+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824277963 0 0) 0x7f7374144b70 con 0x7f73740705b0 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.037+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7374143890 con 0x7f73740705b0 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.037+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7364001930 con 0x7f73740705b0 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.037+0000 7f735bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.037+0000 7f735bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.037+0000 7f735bfff640 10 monclient(hunting): dump: 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:04.039 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:04.040 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:04.040 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:04.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.038+0000 7f73737fe640 1 -- 172.21.15.60:0/2355779356 >> v1:172.21.15.60:6790/0 conn(0x7f73740705b0 legacy=0x7f7374134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:28:04.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.038+0000 7f73737fe640 1 -- 172.21.15.60:0/2355779356 >> v1:172.21.15.60:6790/0 conn(0x7f73740705b0 legacy=0x7f7374134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:28:04.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.038+0000 7f73737fe640 1 --1- 172.21.15.60:0/2355779356 >> v1:172.21.15.60:6790/0 conn(0x7f73740705b0 0x7f7374134140 :-1 s=OPENED pgs=57343 cs=1 l=1).handle_message read tag failed 2024-03-25T19:28:04.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.038+0000 7f73737fe640 1 --1- 172.21.15.60:0/2355779356 >> v1:172.21.15.60:6790/0 conn(0x7f73740705b0 0x7f7374134140 :-1 s=OPENED pgs=57343 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:28:04.040 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.038+0000 7f735bfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.60:6790/0 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1244666852 0 0) 0x7f7374143370 con 0x7f73741402d0 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 >> v1:172.21.15.60:6790/0 conn(0x7f73740705b0 legacy=0x7f7374134140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 >> v1:172.21.15.60:6789/0 conn(0x7f7374149ae0 legacy=0x7f73741e1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 1 monclient: found mon.b 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7374144560 con 0x7f73741402d0 2024-03-25T19:28:04.064 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 10 monclient: _finish_auth 0 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f735bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 10 monclient: _renew_subs 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7374143000 con 0x7f73741402d0 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 10 monclient: _renew_subs 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73740af930 con 0x7f73741402d0 2024-03-25T19:28:04.065 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.063+0000 7f7379f04640 1 librados: init done 2024-03-25T19:28:04.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.064+0000 7f7379f04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:04.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.064+0000 7f7379f04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:04.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.064+0000 7f7379f04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:04.066 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.064+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73740627b0 con 0x7f73741402d0 2024-03-25T19:28:04.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.364+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7368001c70 con 0x7f73741402d0 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.364+0000 7f735bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.364+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7368003a60 con 0x7f73741402d0 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.364+0000 7f735bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.364+0000 7f735bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.364+0000 7f735bfff640 10 monclient: dump: 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:04.366 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:04.367 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:04.367 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:04.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.364+0000 7f7371ffb640 4 set_mon_vals no callback set 2024-03-25T19:28:04.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.365+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f7368003c40 con 0x7f73741402d0 2024-03-25T19:28:04.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.366+0000 7f73737fe640 10 monclient: get_auth_request con 0x7f736006dfc0 auth_method 0 2024-03-25T19:28:04.367 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.366+0000 7f735bfff640 10 client.4564.objecter ms_handle_connect 0x7f736006dfc0 2024-03-25T19:28:04.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.370+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f73680548a0 con 0x7f73741402d0 2024-03-25T19:28:04.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.370+0000 7f735bfff640 10 client.4564.objecter ms_dispatch 0x7f7374000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:04.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.370+0000 7f735bfff640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:04.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.370+0000 7f735bfff640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:04.371 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.370+0000 7f735bfff640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T19:28:04.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.372+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73680503b0 con 0x7f73741402d0 2024-03-25T19:28:04.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.372+0000 7f735bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:04.372 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.372+0000 7f735bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:04.633 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.632+0000 7f7379f04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:04.633 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.632+0000 7f7379f04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:04.633 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.632+0000 7f7379f04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:04.633 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.632+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7374143190 con 0x7f73741402d0 2024-03-25T19:28:04.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.663+0000 7f735bfff640 1 -- 172.21.15.60:0/2355779356 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f73680522e0 con 0x7f73741402d0 2024-03-25T19:28:04.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.663+0000 7f735bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:04.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.663+0000 7f735bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:04.664 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:04.664 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:04.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 10 librados: watch_flush enter 2024-03-25T19:28:04.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 10 librados: watch_flush exit 2024-03-25T19:28:04.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T19:28:04.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 10 client.4564.objecter successfully canceled tick 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 >> v1:172.21.15.90:6800/1827638754 conn(0x7f736006dfc0 legacy=0x7f7360070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 10 monclient: shutdown 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 >> v1:172.21.15.90:6789/0 conn(0x7f73741402d0 legacy=0x7f7374141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 shutdown_connections 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 >> 172.21.15.60:0/2355779356 conn(0x7f73740b5d10 msgr2=0x7f73740b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.665+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 shutdown_connections 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.666+0000 7f7379f04640 1 -- 172.21.15.60:0/2355779356 wait complete. 2024-03-25T19:28:04.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:04.666+0000 7f7379f04640 1 librados: shutdown 2024-03-25T19:28:04.677 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:11.483 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.595+0000 7f2749172640 10 monclient: get_monmap_and_config 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.595+0000 7f2749172640 10 monclient: build_initial_monmap 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.595+0000 7f2749172640 10 monclient: monmap: 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:11.597037+0000 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:11.597037+0000 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:11.596 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:11.597 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:11.597 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:11.597 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:11.597 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:11.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 1 Processor -- start 2024-03-25T19:28:11.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 1 -- start start 2024-03-25T19:28:11.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient: init 2024-03-25T19:28:11.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:11.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:11.598 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient(hunting): picked mon.noname-b con 0x7f27440a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:11.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient(hunting): picked mon.noname-c con 0x7f274413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:11.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient(hunting): picked mon.noname-a con 0x7f27441430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:11.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2744142520 con 0x7f27441430c0 2024-03-25T19:28:11.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27440b28a0 con 0x7f274413f710 2024-03-25T19:28:11.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27440b2a30 con 0x7f27440a98f0 2024-03-25T19:28:11.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:11.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.596+0000 7f2749172640 10 monclient(hunting): authenticate will time out at 6818.695846s 2024-03-25T19:28:11.601 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.600+0000 7f2742d9d640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f27440a98f0 0x7f27440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:51238/0 (socket says 172.21.15.60:51238) 2024-03-25T19:28:11.601 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.600+0000 7f2742d9d640 1 -- 172.21.15.60:0/162544138 learned_addr learned my addr 172.21.15.60:0/162544138 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:11.614 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.614+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260099809 0 0) 0x7f27440b28a0 con 0x7f274413f710 2024-03-25T19:28:11.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.614+0000 7f2741d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:11.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.614+0000 7f2741d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:11.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.614+0000 7f2741d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:11.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.614+0000 7f2741d9b640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:28:11.615 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.614+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f272c003520 con 0x7f274413f710 2024-03-25T19:28:11.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.694+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607511009 0 0) 0x7f27440b2a30 con 0x7f27440a98f0 2024-03-25T19:28:11.694 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.694+0000 7f2741d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:11.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.694+0000 7f2741d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:11.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.694+0000 7f2741d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:11.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.694+0000 7f2741d9b640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:28:11.695 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.694+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27440b28a0 con 0x7f27440a98f0 2024-03-25T19:28:11.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.694+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454912653 0 0) 0x7f272c003520 con 0x7f274413f710 2024-03-25T19:28:11.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 >> v1:172.21.15.60:6790/0 conn(0x7f27440a98f0 legacy=0x7f27440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:11.696 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 >> v1:172.21.15.60:6789/0 conn(0x7f27441430c0 legacy=0x7f274414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 1 monclient: found mon.noname-c 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27440b44c0 con 0x7f274413f710 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 10 monclient: _finish_auth 0 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2738001690 con 0x7f274413f710 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2741d9b640 10 monclient: dump: 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:11.697 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2749172640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:28:11.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.695+0000 7f2749172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27380017f0 con 0x7f274413f710 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2741d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2741d9b640 1 -- 172.21.15.60:0/162544138 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2738001c00 con 0x7f274413f710 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2741d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2741d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2741d9b640 10 monclient: dump: 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:11.714 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2749172640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:11.715 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.713+0000 7f2749172640 4 set_mon_vals no callback set 2024-03-25T19:28:11.716 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.714+0000 7f2749172640 10 monclient: shutdown 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.714+0000 7f2749172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.714+0000 7f2749172640 1 -- 172.21.15.60:0/162544138 >> v1:172.21.15.90:6789/0 conn(0x7f274413f710 legacy=0x7f27441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.714+0000 7f2749172640 1 -- 172.21.15.60:0/162544138 shutdown_connections 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.714+0000 7f2749172640 1 -- 172.21.15.60:0/162544138 >> 172.21.15.60:0/162544138 conn(0x7f27440b7ea0 msgr2=0x7f27440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.714+0000 7f2749172640 1 -- 172.21.15.60:0/162544138 shutdown_connections 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.714+0000 7f2749172640 1 -- 172.21.15.60:0/162544138 wait complete. 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.715+0000 7f2749172640 10 monclient: build_initial_monmap 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.715+0000 7f2749172640 10 monclient: monmap: 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:11.717 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.716+0000 7f2749172640 1 librados: starting msgr at 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.716+0000 7f2749172640 1 librados: starting objecter 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.716+0000 7f2749172640 1 Processor -- start 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.716+0000 7f2749172640 1 -- start start 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.716+0000 7f2749172640 1 librados: setting wanted keys 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.716+0000 7f2749172640 1 librados: calling monclient init 2024-03-25T19:28:11.718 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.716+0000 7f2749172640 10 monclient: init 2024-03-25T19:28:11.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:11.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:11.719 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 10 monclient(hunting): picked mon.noname-c con 0x7f27440a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:11.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 10 monclient(hunting): picked mon.noname-b con 0x7f274413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:11.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 10 monclient(hunting): picked mon.noname-a con 0x7f27441430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:11.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27441d77a0 con 0x7f27441430c0 2024-03-25T19:28:11.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2744138e40 con 0x7f274413f710 2024-03-25T19:28:11.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27440aba30 con 0x7f27440a98f0 2024-03-25T19:28:11.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:11.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.717+0000 7f2749172640 10 monclient(hunting): authenticate will time out at 6818.816430s 2024-03-25T19:28:11.725 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.724+0000 7f274259c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f274413f710 0x7f27441d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:44246/0 (socket says 172.21.15.60:44246) 2024-03-25T19:28:11.725 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.724+0000 7f274259c640 1 -- 172.21.15.60:0/543728476 learned_addr learned my addr 172.21.15.60:0/543728476 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:11.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.725+0000 7f272b7fe640 10 client.?.objecter ms_handle_connect 0x7f274413f710 2024-03-25T19:28:11.726 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.725+0000 7f272b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:11.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.729+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772726096 0 0) 0x7f2744138e40 con 0x7f274413f710 2024-03-25T19:28:11.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.729+0000 7f272b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:11.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.729+0000 7f272b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:11.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.729+0000 7f272b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:11.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.729+0000 7f272b7fe640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:28:11.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.729+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2730004eb0 con 0x7f274413f710 2024-03-25T19:28:11.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.737+0000 7f272b7fe640 10 client.?.objecter ms_handle_connect 0x7f27440a98f0 2024-03-25T19:28:11.738 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.737+0000 7f272b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:11.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.740+0000 7f272b7fe640 10 client.?.objecter ms_handle_connect 0x7f27441430c0 2024-03-25T19:28:11.740 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.740+0000 7f272b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.745+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2992564878 0 0) 0x7f2730004eb0 con 0x7f274413f710 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.745+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2744138e40 con 0x7f274413f710 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.745+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2738001bc0 con 0x7f274413f710 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.745+0000 7f272b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.745+0000 7f272b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.745+0000 7f272b7fe640 10 monclient(hunting): dump: 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:11.746 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:11.747 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:11.747 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:11.747 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:11.747 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:11.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.745+0000 7f272b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:11.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.746+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828356716 0 0) 0x7f27440aba30 con 0x7f27440a98f0 2024-03-25T19:28:11.747 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.746+0000 7f272b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:11.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.746+0000 7f272b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:11.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.746+0000 7f272b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:11.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.746+0000 7f272b7fe640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:28:11.748 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.746+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2730004eb0 con 0x7f27440a98f0 2024-03-25T19:28:11.761 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.761+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490503851 0 0) 0x7f27441d77a0 con 0x7f27441430c0 2024-03-25T19:28:11.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.761+0000 7f272b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:11.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.761+0000 7f272b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:11.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.761+0000 7f272b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:11.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.761+0000 7f272b7fe640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:28:11.762 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.761+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27440aba30 con 0x7f27441430c0 2024-03-25T19:28:11.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 661393346 0 0) 0x7f2744138e40 con 0x7f274413f710 2024-03-25T19:28:11.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 >> v1:172.21.15.60:6790/0 conn(0x7f27440a98f0 legacy=0x7f2744138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:11.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 >> v1:172.21.15.60:6789/0 conn(0x7f27441430c0 legacy=0x7f2744134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:11.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:11.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 1 monclient: found mon.b 2024-03-25T19:28:11.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:11.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27441d8340 con 0x7f274413f710 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f272b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 10 monclient: _renew_subs 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2744134820 con 0x7f274413f710 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 10 monclient: _renew_subs 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27441380c0 con 0x7f274413f710 2024-03-25T19:28:11.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.772+0000 7f2749172640 1 librados: init done 2024-03-25T19:28:11.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.773+0000 7f2749172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:11.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.773+0000 7f2749172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:11.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.773+0000 7f2749172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:11.775 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:11.773+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27440627b0 con 0x7f274413f710 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.051+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27380011f0 con 0x7f274413f710 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.051+0000 7f272b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.051+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2738003a40 con 0x7f274413f710 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.051+0000 7f272b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.051+0000 7f272b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.051+0000 7f272b7fe640 10 monclient: dump: 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:12.052 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:12.053 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.051+0000 7f274159a640 4 set_mon_vals no callback set 2024-03-25T19:28:12.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.052+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2738003c60 con 0x7f274413f710 2024-03-25T19:28:12.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.053+0000 7f2742d9d640 10 monclient: get_auth_request con 0x7f273006de00 auth_method 0 2024-03-25T19:28:12.054 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.053+0000 7f272b7fe640 10 client.4570.objecter ms_handle_connect 0x7f273006de00 2024-03-25T19:28:12.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f2738054970 con 0x7f274413f710 2024-03-25T19:28:12.057 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 10 client.4570.objecter ms_dispatch 0x7f2744000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:12.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:12.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:12.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T19:28:12.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27380502b0 con 0x7f274413f710 2024-03-25T19:28:12.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:12.058 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.057+0000 7f272b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:12.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.311+0000 7f2749172640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:12.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.311+0000 7f2749172640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:12.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.311+0000 7f2749172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:12.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.311+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f274414e8b0 con 0x7f274413f710 2024-03-25T19:28:12.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.342+0000 7f272b7fe640 1 -- 172.21.15.60:0/543728476 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f27380523b0 con 0x7f274413f710 2024-03-25T19:28:12.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.342+0000 7f272b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:12.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.342+0000 7f272b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:12.343 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:12.343 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:12.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 10 librados: watch_flush enter 2024-03-25T19:28:12.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 10 librados: watch_flush exit 2024-03-25T19:28:12.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T19:28:12.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 10 client.4570.objecter successfully canceled tick 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 >> v1:172.21.15.90:6800/1827638754 conn(0x7f273006de00 legacy=0x7f27300702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 10 monclient: shutdown 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 >> v1:172.21.15.90:6789/0 conn(0x7f274413f710 legacy=0x7f27441d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 shutdown_connections 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 >> 172.21.15.60:0/543728476 conn(0x7f27440b7ea0 msgr2=0x7f2744074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 shutdown_connections 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 1 -- 172.21.15.60:0/543728476 wait complete. 2024-03-25T19:28:12.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:12.344+0000 7f2749172640 1 librados: shutdown 2024-03-25T19:28:12.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:19.164 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:19.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.277+0000 7fc3076ff640 10 monclient: get_monmap_and_config 2024-03-25T19:28:19.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.277+0000 7fc3076ff640 10 monclient: build_initial_monmap 2024-03-25T19:28:19.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.277+0000 7fc3076ff640 10 monclient: monmap: 2024-03-25T19:28:19.278 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:19.278 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:19.279435+0000 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:19.279435+0000 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:19.279 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:19.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.278+0000 7fc3076ff640 1 Processor -- start 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.278+0000 7fc3076ff640 1 -- start start 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.278+0000 7fc3076ff640 10 monclient: init 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 10 monclient(hunting): picked mon.noname-a con 0x7fc3000a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 10 monclient(hunting): picked mon.noname-c con 0x7fc30013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 10 monclient(hunting): picked mon.noname-b con 0x7fc3001430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc300142520 con 0x7fc3000a98f0 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3000b28a0 con 0x7fc30013f710 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3000b2a30 con 0x7fc3001430c0 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:19.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.279+0000 7fc3076ff640 10 monclient(hunting): authenticate will time out at 6826.378524s 2024-03-25T19:28:19.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.296+0000 7fc304c9a640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fc30013f710 0x7fc3001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:54430/0 (socket says 172.21.15.60:54430) 2024-03-25T19:28:19.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.296+0000 7fc304c9a640 1 -- 172.21.15.60:0/1665534911 learned_addr learned my addr 172.21.15.60:0/1665534911 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:19.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.315+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461589801 0 0) 0x7fc3000b28a0 con 0x7fc30013f710 2024-03-25T19:28:19.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.315+0000 7fc2f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:19.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.315+0000 7fc2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:19.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.315+0000 7fc2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:19.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.315+0000 7fc2f3fff640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:28:19.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.315+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2ec003520 con 0x7fc30013f710 2024-03-25T19:28:19.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.322+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195874588 0 0) 0x7fc3000b2a30 con 0x7fc3001430c0 2024-03-25T19:28:19.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.322+0000 7fc2f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:19.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.322+0000 7fc2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:19.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.322+0000 7fc2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:19.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.322+0000 7fc2f3fff640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:28:19.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.322+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3000b28a0 con 0x7fc3001430c0 2024-03-25T19:28:19.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939638310 0 0) 0x7fc300142520 con 0x7fc3000a98f0 2024-03-25T19:28:19.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:19.337 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3000b2a30 con 0x7fc3000a98f0 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3121692867 0 0) 0x7fc2ec003520 con 0x7fc30013f710 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 >> v1:172.21.15.60:6790/0 conn(0x7fc3001430c0 legacy=0x7fc30014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.336+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 >> v1:172.21.15.60:6789/0 conn(0x7fc3000a98f0 legacy=0x7fc3000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 1 monclient: found mon.noname-c 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3000b44c0 con 0x7fc30013f710 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:19.338 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc2fc001690 con 0x7fc30013f710 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc3076ff640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc3076ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc2f3fff640 10 monclient: dump: 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.339 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:19.340 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:19.340 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:19.340 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:19.340 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:19.340 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:19.340 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.337+0000 7fc3076ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2fc0017f0 con 0x7fc30013f710 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc2f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc2f3fff640 1 -- 172.21.15.60:0/1665534911 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc2fc001c00 con 0x7fc30013f710 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc2f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc2f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc2f3fff640 10 monclient: dump: 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:19.344 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc3076ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:19.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.343+0000 7fc3076ff640 4 set_mon_vals no callback set 2024-03-25T19:28:19.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.344+0000 7fc3076ff640 10 monclient: shutdown 2024-03-25T19:28:19.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.344+0000 7fc3076ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:19.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.344+0000 7fc3076ff640 1 -- 172.21.15.60:0/1665534911 >> v1:172.21.15.90:6789/0 conn(0x7fc30013f710 legacy=0x7fc3001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.344+0000 7fc3076ff640 1 -- 172.21.15.60:0/1665534911 shutdown_connections 2024-03-25T19:28:19.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.344+0000 7fc3076ff640 1 -- 172.21.15.60:0/1665534911 >> 172.21.15.60:0/1665534911 conn(0x7fc3000b7ea0 msgr2=0x7fc3000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:19.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.344+0000 7fc3076ff640 1 -- 172.21.15.60:0/1665534911 shutdown_connections 2024-03-25T19:28:19.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.344+0000 7fc3076ff640 1 -- 172.21.15.60:0/1665534911 wait complete. 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.345+0000 7fc3076ff640 10 monclient: build_initial_monmap 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.345+0000 7fc3076ff640 10 monclient: monmap: 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:19.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.345+0000 7fc3076ff640 1 librados: starting msgr at 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.345+0000 7fc3076ff640 1 librados: starting objecter 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 1 Processor -- start 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 1 -- start start 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 1 librados: setting wanted keys 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 1 librados: calling monclient init 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient: init 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient(hunting): picked mon.noname-c con 0x7fc3000a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient(hunting): picked mon.noname-b con 0x7fc30013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient(hunting): picked mon.noname-a con 0x7fc3001430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc300138a70 con 0x7fc3001430c0 2024-03-25T19:28:19.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3000aba50 con 0x7fc30013f710 2024-03-25T19:28:19.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3001d77a0 con 0x7fc3000a98f0 2024-03-25T19:28:19.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:19.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.346+0000 7fc3076ff640 10 monclient(hunting): authenticate will time out at 6826.445775s 2024-03-25T19:28:19.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.348+0000 7fc305c9c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fc3001430c0 0x7fc3001d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:54368/0 (socket says 172.21.15.60:54368) 2024-03-25T19:28:19.350 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.348+0000 7fc305c9c640 1 -- 172.21.15.60:0/590996918 learned_addr learned my addr 172.21.15.60:0/590996918 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:19.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.350+0000 7fc2f1ffb640 10 client.?.objecter ms_handle_connect 0x7fc3001430c0 2024-03-25T19:28:19.351 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.350+0000 7fc2f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:19.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.358+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173555729 0 0) 0x7fc300138a70 con 0x7fc3001430c0 2024-03-25T19:28:19.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.359+0000 7fc2f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:19.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.359+0000 7fc2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:19.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.359+0000 7fc2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:19.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.359+0000 7fc2f1ffb640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:28:19.359 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.359+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e4004eb0 con 0x7fc3001430c0 2024-03-25T19:28:19.362 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.361+0000 7fc2f1ffb640 10 client.?.objecter ms_handle_connect 0x7fc30013f710 2024-03-25T19:28:19.362 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.361+0000 7fc2f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478342787 0 0) 0x7fc3000aba50 con 0x7fc30013f710 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc300138a70 con 0x7fc30013f710 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 10 client.?.objecter ms_handle_connect 0x7fc3000a98f0 2024-03-25T19:28:19.370 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.369+0000 7fc2f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:19.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.378+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1534289537 0 0) 0x7fc300138a70 con 0x7fc30013f710 2024-03-25T19:28:19.378 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.378+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3000aba50 con 0x7fc30013f710 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.378+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc2fc001b00 con 0x7fc30013f710 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.378+0000 7fc2f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.378+0000 7fc2f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.378+0000 7fc2f1ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:19.379 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:19.380 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.378+0000 7fc2f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.382+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2359113113 0 0) 0x7fc2e4004eb0 con 0x7fc3001430c0 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.382+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc300138a70 con 0x7fc3001430c0 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.382+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc2f8001c90 con 0x7fc3001430c0 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.382+0000 7fc2f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.382+0000 7fc2f1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.382+0000 7fc2f1ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:19.383 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:19.384 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:19.384 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:19.384 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:19.384 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:19.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.384+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181753813 0 0) 0x7fc3001d77a0 con 0x7fc3000a98f0 2024-03-25T19:28:19.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.384+0000 7fc2f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:19.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.384+0000 7fc2f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:19.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.384+0000 7fc2f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:19.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.384+0000 7fc2f1ffb640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:28:19.385 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.384+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e4004eb0 con 0x7fc3000a98f0 2024-03-25T19:28:19.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1974587711 0 0) 0x7fc300138a70 con 0x7fc3001430c0 2024-03-25T19:28:19.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 >> v1:172.21.15.60:6790/0 conn(0x7fc3000a98f0 legacy=0x7fc300138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.388 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 >> v1:172.21.15.90:6789/0 conn(0x7fc30013f710 legacy=0x7fc3001d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 1 monclient: found mon.a 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3001d83d0 con 0x7fc3001430c0 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc2f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc3076ff640 5 monclient: authenticate success, global_id 4587 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc3076ff640 10 monclient: _renew_subs 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc3076ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.387+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3001d41e0 con 0x7fc3001430c0 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.388+0000 7fc3076ff640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:19.389 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.388+0000 7fc3076ff640 10 monclient: _renew_subs 2024-03-25T19:28:19.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.388+0000 7fc3076ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:19.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.388+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3001d5110 con 0x7fc3001430c0 2024-03-25T19:28:19.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.388+0000 7fc3076ff640 1 librados: init done 2024-03-25T19:28:19.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.389+0000 7fc3076ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:19.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.389+0000 7fc3076ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:19.390 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.389+0000 7fc3076ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:19.391 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.389+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc30014f320 con 0x7fc3001430c0 2024-03-25T19:28:19.689 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.688+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2f8003970 con 0x7fc3001430c0 2024-03-25T19:28:19.689 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.688+0000 7fc2f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.688+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fc2f8003d40 con 0x7fc3001430c0 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.688+0000 7fc2f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.689+0000 7fc2f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.689+0000 7fc2f1ffb640 10 monclient: dump: 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:19.690 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:19.691 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:19.691 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:19.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.689+0000 7fc2f37fe640 4 set_mon_vals no callback set 2024-03-25T19:28:19.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.689+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fc2f8003f60 con 0x7fc3001430c0 2024-03-25T19:28:19.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.690+0000 7fc30549b640 10 monclient: get_auth_request con 0x7fc2e406df50 auth_method 0 2024-03-25T19:28:19.691 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.690+0000 7fc2f1ffb640 10 client.4587.objecter ms_handle_connect 0x7fc2e406df50 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fc2f8054ae0 con 0x7fc3001430c0 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 10 client.4587.objecter ms_dispatch 0x7fc300000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2f80505f0 con 0x7fc3001430c0 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:19.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.698+0000 7fc2f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:19.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.919+0000 7fc3076ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:19.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.919+0000 7fc3076ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:19.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.919+0000 7fc3076ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:19.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.919+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3001d7930 con 0x7fc3001430c0 2024-03-25T19:28:19.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.944+0000 7fc2f1ffb640 1 -- 172.21.15.60:0/590996918 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fc2f8052520 con 0x7fc3001430c0 2024-03-25T19:28:19.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.944+0000 7fc2f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:19.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.944+0000 7fc2f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:19.945 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:19.946 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:19.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 10 librados: watch_flush enter 2024-03-25T19:28:19.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 10 librados: watch_flush exit 2024-03-25T19:28:19.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T19:28:19.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 10 client.4587.objecter successfully canceled tick 2024-03-25T19:28:19.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 >> v1:172.21.15.90:6800/1827638754 conn(0x7fc2e406df50 legacy=0x7fc2e4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 10 monclient: shutdown 2024-03-25T19:28:19.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:19.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 >> v1:172.21.15.60:6789/0 conn(0x7fc3001430c0 legacy=0x7fc3001d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:19.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 shutdown_connections 2024-03-25T19:28:19.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 >> 172.21.15.60:0/590996918 conn(0x7fc3000b7ea0 msgr2=0x7fc30013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:19.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 shutdown_connections 2024-03-25T19:28:19.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 1 -- 172.21.15.60:0/590996918 wait complete. 2024-03-25T19:28:19.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:19.947+0000 7fc3076ff640 1 librados: shutdown 2024-03-25T19:28:19.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:20.594 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:21.305 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.603 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:21.315 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.631 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:21.342 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.649 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:21.303 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.653 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:21.307 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.666 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:21.377 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.674 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:21.385 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.674 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:21.328 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:20.714 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:21.425 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:26.765 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:26.875 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.875+0000 7fe288f67640 10 monclient: get_monmap_and_config 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.875+0000 7fe288f67640 10 monclient: build_initial_monmap 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.875+0000 7fe288f67640 10 monclient: monmap: 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:26.877008+0000 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:26.877008+0000 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:26.876 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:26.877 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:26.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 1 Processor -- start 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 1 -- start start 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient: init 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient(hunting): picked mon.noname-c con 0x7fe2840b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient(hunting): picked mon.noname-b con 0x7fe2840b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient(hunting): picked mon.noname-a con 0x7fe28414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840af940 con 0x7fe28414be90 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840b0a70 con 0x7fe2840b4d60 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2840b0d50 con 0x7fe2840b1730 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.876+0000 7fe288f67640 10 monclient(hunting): authenticate will time out at 6833.975786s 2024-03-25T19:28:26.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.877+0000 7fe28259c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fe2840b4d60 0x7fe28413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:34764/0 (socket says 172.21.15.60:34764) 2024-03-25T19:28:26.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.877+0000 7fe28259c640 1 -- 172.21.15.60:0/4136706573 learned_addr learned my addr 172.21.15.60:0/4136706573 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:26.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.889+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378010157 0 0) 0x7fe2840b0a70 con 0x7fe2840b4d60 2024-03-25T19:28:26.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.889+0000 7fe281d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.889+0000 7fe281d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.889+0000 7fe281d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.889+0000 7fe281d9b640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:28:26.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.889+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270003520 con 0x7fe2840b4d60 2024-03-25T19:28:26.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.899+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685668131 0 0) 0x7fe2840af940 con 0x7fe28414be90 2024-03-25T19:28:26.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.899+0000 7fe281d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.899+0000 7fe281d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.899+0000 7fe281d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.899+0000 7fe281d9b640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:28:26.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.899+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2840b0a70 con 0x7fe28414be90 2024-03-25T19:28:26.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199383279 0 0) 0x7fe270003520 con 0x7fe2840b4d60 2024-03-25T19:28:26.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 >> v1:172.21.15.60:6790/0 conn(0x7fe2840b1730 legacy=0x7fe2840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 >> v1:172.21.15.60:6789/0 conn(0x7fe28414be90 legacy=0x7fe28414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 1 monclient: found mon.noname-c 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.905+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2840b1030 con 0x7fe2840b4d60 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe281d9b640 10 monclient: _finish_auth 0 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe281d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe281d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe274001690 con 0x7fe2840b4d60 2024-03-25T19:28:26.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe281d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe281d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe281d9b640 10 monclient: dump: 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe288f67640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:28:26.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.906+0000 7fe288f67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:26.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.917+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2740017f0 con 0x7fe2840b4d60 2024-03-25T19:28:26.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe281d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:26.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe281d9b640 1 -- 172.21.15.60:0/4136706573 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe274001c00 con 0x7fe2840b4d60 2024-03-25T19:28:26.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe281d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe281d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe281d9b640 10 monclient: dump: 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe288f67640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:26.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe288f67640 4 set_mon_vals no callback set 2024-03-25T19:28:26.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe288f67640 10 monclient: shutdown 2024-03-25T19:28:26.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe288f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:26.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.918+0000 7fe288f67640 1 -- 172.21.15.60:0/4136706573 >> v1:172.21.15.90:6789/0 conn(0x7fe2840b4d60 legacy=0x7fe28413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.919+0000 7fe288f67640 1 -- 172.21.15.60:0/4136706573 shutdown_connections 2024-03-25T19:28:26.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.919+0000 7fe288f67640 1 -- 172.21.15.60:0/4136706573 >> 172.21.15.60:0/4136706573 conn(0x7fe2840617a0 msgr2=0x7fe284061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:26.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.919+0000 7fe288f67640 1 -- 172.21.15.60:0/4136706573 shutdown_connections 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.919+0000 7fe288f67640 1 -- 172.21.15.60:0/4136706573 wait complete. 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 10 monclient: build_initial_monmap 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 10 monclient: monmap: 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:26.921 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:26.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 1 librados: starting msgr at 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 1 librados: starting objecter 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 1 Processor -- start 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 1 -- start start 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 1 librados: setting wanted keys 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 1 librados: calling monclient init 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.920+0000 7fe288f67640 10 monclient: init 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 10 monclient(hunting): picked mon.noname-b con 0x7fe2840b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 10 monclient(hunting): picked mon.noname-c con 0x7fe2840b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 10 monclient(hunting): picked mon.noname-a con 0x7fe28414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2841dba90 con 0x7fe28414be90 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2841dbc20 con 0x7fe2840b1730 2024-03-25T19:28:26.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe28413b560 con 0x7fe2840b4d60 2024-03-25T19:28:26.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:26.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.921+0000 7fe288f67640 10 monclient(hunting): authenticate will time out at 6834.020473s 2024-03-25T19:28:26.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.933+0000 7fe27bfff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fe2840b4d60 0x7fe284142d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:36910/0 (socket says 172.21.15.60:36910) 2024-03-25T19:28:26.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.933+0000 7fe27bfff640 1 -- 172.21.15.60:0/2756660792 learned_addr learned my addr 172.21.15.60:0/2756660792 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:26.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.934+0000 7fe27affd640 10 client.?.objecter ms_handle_connect 0x7fe2840b4d60 2024-03-25T19:28:26.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.934+0000 7fe27affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.937+0000 7fe27affd640 10 client.?.objecter ms_handle_connect 0x7fe28414be90 2024-03-25T19:28:26.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.937+0000 7fe27affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.938+0000 7fe27affd640 10 client.?.objecter ms_handle_connect 0x7fe2840b1730 2024-03-25T19:28:26.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.938+0000 7fe27affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:26.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.941+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590107151 0 0) 0x7fe28413b560 con 0x7fe2840b4d60 2024-03-25T19:28:26.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.941+0000 7fe27affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.941+0000 7fe27affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.941+0000 7fe27affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.941+0000 7fe27affd640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:28:26.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.941+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe264004eb0 con 0x7fe2840b4d60 2024-03-25T19:28:26.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.948+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412379229 0 0) 0x7fe2841dba90 con 0x7fe28414be90 2024-03-25T19:28:26.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.948+0000 7fe27affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.948+0000 7fe27affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.948+0000 7fe27affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.948+0000 7fe27affd640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:28:26.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.948+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28413b560 con 0x7fe28414be90 2024-03-25T19:28:26.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.951+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3245175269 0 0) 0x7fe264004eb0 con 0x7fe2840b4d60 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.951+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2841dba90 con 0x7fe2840b4d60 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.951+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe26c001a30 con 0x7fe2840b4d60 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.951+0000 7fe27affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.951+0000 7fe27affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.951+0000 7fe27affd640 10 monclient(hunting): dump: 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:26.952 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:26.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.951+0000 7fe27affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:26.956 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.956+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279753494 0 0) 0x7fe2841dbc20 con 0x7fe2840b1730 2024-03-25T19:28:26.956 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.956+0000 7fe27affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:26.956 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.956+0000 7fe27affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:26.956 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.956+0000 7fe27affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:26.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.956+0000 7fe27affd640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:28:26.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.956+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe264004eb0 con 0x7fe2840b1730 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3394337771 0 0) 0x7fe2841dba90 con 0x7fe2840b4d60 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 >> v1:172.21.15.90:6789/0 conn(0x7fe2840b1730 legacy=0x7fe2841db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 >> v1:172.21.15.60:6789/0 conn(0x7fe28414be90 legacy=0x7fe2841405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 1 monclient: found mon.c 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28413b6f0 con 0x7fe2840b4d60 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe27affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 5 monclient: authenticate success, global_id 4580 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 10 monclient: _renew_subs 2024-03-25T19:28:26.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe284143430 con 0x7fe2840b4d60 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 10 monclient: _renew_subs 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe284142610 con 0x7fe2840b4d60 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.959+0000 7fe288f67640 1 librados: init done 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.960+0000 7fe288f67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.960+0000 7fe288f67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:26.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.960+0000 7fe288f67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:26.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:26.960+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2840627b0 con 0x7fe2840b4d60 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.286+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe26c001c70 con 0x7fe2840b4d60 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.286+0000 7fe27affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.286+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fe26c003a60 con 0x7fe2840b4d60 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.286+0000 7fe27affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.286+0000 7fe27affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.286+0000 7fe27affd640 10 monclient: dump: 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:27.287 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:27.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.286+0000 7fe28159a640 4 set_mon_vals no callback set 2024-03-25T19:28:27.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.287+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fe26c003c80 con 0x7fe2840b4d60 2024-03-25T19:28:27.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.288+0000 7fe28259c640 10 monclient: get_auth_request con 0x7fe26406de00 auth_method 0 2024-03-25T19:28:27.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.288+0000 7fe27affd640 10 client.4580.objecter ms_handle_connect 0x7fe26406de00 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fe26c0548a0 con 0x7fe2840b4d60 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 10 client.4580.objecter ms_dispatch 0x7fe284000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe26c0503b0 con 0x7fe2840b4d60 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:27.298 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.295+0000 7fe27affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:27.544 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.543+0000 7fe288f67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:27.544 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.543+0000 7fe288f67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:27.544 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.543+0000 7fe288f67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:27.545 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.543+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2841daf80 con 0x7fe2840b4d60 2024-03-25T19:28:27.574 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.574+0000 7fe27affd640 1 -- 172.21.15.60:0/2756660792 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fe26c0522e0 con 0x7fe2840b4d60 2024-03-25T19:28:27.574 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.574+0000 7fe27affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:27.574 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.574+0000 7fe27affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:27.575 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:27.575 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:27.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 10 librados: watch_flush enter 2024-03-25T19:28:27.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 10 librados: watch_flush exit 2024-03-25T19:28:27.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-25T19:28:27.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 10 client.4580.objecter successfully canceled tick 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 >> v1:172.21.15.90:6800/1827638754 conn(0x7fe26406de00 legacy=0x7fe2640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 10 monclient: shutdown 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 >> v1:172.21.15.60:6790/0 conn(0x7fe2840b4d60 legacy=0x7fe284142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 shutdown_connections 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 >> 172.21.15.60:0/2756660792 conn(0x7fe2840617a0 msgr2=0x7fe2840b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 shutdown_connections 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 1 -- 172.21.15.60:0/2756660792 wait complete. 2024-03-25T19:28:27.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:27.576+0000 7fe288f67640 1 librados: shutdown 2024-03-25T19:28:27.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:34.397 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.514+0000 7f93e7493640 10 monclient: get_monmap_and_config 2024-03-25T19:28:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.514+0000 7f93e7493640 10 monclient: build_initial_monmap 2024-03-25T19:28:34.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.514+0000 7f93e7493640 10 monclient: monmap: 2024-03-25T19:28:34.514 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:34.515485+0000 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:34.515485+0000 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:34.515 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:34.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.515+0000 7f93e7493640 1 Processor -- start 2024-03-25T19:28:34.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.515+0000 7f93e7493640 1 -- start start 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.515+0000 7f93e7493640 10 monclient: init 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 10 monclient(hunting): picked mon.noname-b con 0x7f93e00b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 10 monclient(hunting): picked mon.noname-c con 0x7f93e00b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 10 monclient(hunting): picked mon.noname-a con 0x7f93e014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e00af940 con 0x7f93e014be90 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e00b0a70 con 0x7f93e00b1730 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e00b0d50 con 0x7f93e00b4d60 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:34.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.516+0000 7f93e7493640 10 monclient(hunting): authenticate will time out at 6841.614265s 2024-03-25T19:28:34.523 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.523+0000 7f93e522f640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f93e00b4d60 0x7f93e013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:41464/0 (socket says 172.21.15.60:41464) 2024-03-25T19:28:34.523 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.523+0000 7f93e522f640 1 -- 172.21.15.60:0/2819823994 learned_addr learned my addr 172.21.15.60:0/2819823994 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:34.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.565+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323641126 0 0) 0x7f93e00af940 con 0x7f93e014be90 2024-03-25T19:28:34.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.565+0000 7f93d7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:28:34.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.565+0000 7f93d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:34.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.565+0000 7f93d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:34.564 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.565+0000 7f93d7fff640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:28:34.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.565+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93cc003520 con 0x7f93e014be90 2024-03-25T19:28:34.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054240263 0 0) 0x7f93cc003520 con 0x7f93e014be90 2024-03-25T19:28:34.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 >> v1:172.21.15.60:6790/0 conn(0x7f93e00b4d60 legacy=0x7f93e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:34.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 >> v1:172.21.15.90:6789/0 conn(0x7f93e00b1730 legacy=0x7f93e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:34.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:28:34.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 1 monclient: found mon.noname-a 2024-03-25T19:28:34.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:34.651 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e00b1030 con 0x7f93e014be90 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 10 monclient: _finish_auth 0 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f93d8001a10 con 0x7f93e014be90 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93d7fff640 10 monclient: dump: 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:34.652 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:34.653 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:34.653 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:34.653 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:34.653 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:34.653 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:34.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93e7493640 5 monclient: authenticate success, global_id 4596 2024-03-25T19:28:34.653 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.651+0000 7f93e7493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:34.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93d8001b70 con 0x7f93e014be90 2024-03-25T19:28:34.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:34.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93d7fff640 1 -- 172.21.15.60:0/2819823994 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f93d8001fc0 con 0x7f93e014be90 2024-03-25T19:28:34.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:34.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93d7fff640 10 monclient: dump: 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93e7493640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:34.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.660+0000 7f93e7493640 4 set_mon_vals no callback set 2024-03-25T19:28:34.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.661+0000 7f93e7493640 10 monclient: shutdown 2024-03-25T19:28:34.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.661+0000 7f93e7493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:34.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.661+0000 7f93e7493640 1 -- 172.21.15.60:0/2819823994 >> v1:172.21.15.60:6789/0 conn(0x7f93e014be90 legacy=0x7f93e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.661+0000 7f93e7493640 1 -- 172.21.15.60:0/2819823994 shutdown_connections 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.661+0000 7f93e7493640 1 -- 172.21.15.60:0/2819823994 >> 172.21.15.60:0/2819823994 conn(0x7f93e00617a0 msgr2=0x7f93e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.661+0000 7f93e7493640 1 -- 172.21.15.60:0/2819823994 shutdown_connections 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.661+0000 7f93e7493640 1 -- 172.21.15.60:0/2819823994 wait complete. 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 10 monclient: build_initial_monmap 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 10 monclient: monmap: 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:34.664 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 1 librados: starting msgr at 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 1 librados: starting objecter 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 1 Processor -- start 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 1 -- start start 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 1 librados: setting wanted keys 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 1 librados: calling monclient init 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.662+0000 7f93e7493640 10 monclient: init 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 10 monclient(hunting): picked mon.noname-c con 0x7f93e00b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 10 monclient(hunting): picked mon.noname-b con 0x7f93e00b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 10 monclient(hunting): picked mon.noname-a con 0x7f93e014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e0140380 con 0x7f93e014be90 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e0140660 con 0x7f93e00b4d60 2024-03-25T19:28:34.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e0141e30 con 0x7f93e00b1730 2024-03-25T19:28:34.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:34.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.663+0000 7f93e7493640 10 monclient(hunting): authenticate will time out at 6841.761426s 2024-03-25T19:28:34.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.678+0000 7f93e5a30640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f93e014be90 0x7f93e01e2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52424/0 (socket says 172.21.15.60:52424) 2024-03-25T19:28:34.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.678+0000 7f93e5a30640 1 -- 172.21.15.60:0/2594153590 learned_addr learned my addr 172.21.15.60:0/2594153590 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:34.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.679+0000 7f93d5ffb640 10 client.?.objecter ms_handle_connect 0x7f93e014be90 2024-03-25T19:28:34.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.679+0000 7f93d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:34.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.694+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290969407 0 0) 0x7f93e0140380 con 0x7f93e014be90 2024-03-25T19:28:34.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.694+0000 7f93d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:28:34.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.694+0000 7f93d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:34.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.694+0000 7f93d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:34.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.694+0000 7f93d5ffb640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:28:34.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.694+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c4004eb0 con 0x7f93e014be90 2024-03-25T19:28:34.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.699+0000 7f93d5ffb640 10 client.?.objecter ms_handle_connect 0x7f93e00b1730 2024-03-25T19:28:34.698 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.699+0000 7f93d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:34.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.705+0000 7f93d5ffb640 10 client.?.objecter ms_handle_connect 0x7f93e00b4d60 2024-03-25T19:28:34.704 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.705+0000 7f93d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.711+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2487281460 0 0) 0x7f93c4004eb0 con 0x7f93e014be90 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.711+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93e0140380 con 0x7f93e014be90 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.711+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f93d8001fc0 con 0x7f93e014be90 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.711+0000 7f93d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.711+0000 7f93d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.711+0000 7f93d5ffb640 10 monclient(hunting): dump: 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:34.711 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:34.712 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:34.712 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:34.712 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:34.712 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:34.712 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:34.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.711+0000 7f93d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1833635052 0 0) 0x7f93e0140380 con 0x7f93e014be90 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 >> v1:172.21.15.60:6790/0 conn(0x7f93e00b1730 legacy=0x7f93e0138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 >> v1:172.21.15.90:6789/0 conn(0x7f93e00b4d60 legacy=0x7f93e0142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 1 monclient: found mon.a 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e0142110 con 0x7f93e014be90 2024-03-25T19:28:34.720 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 5 monclient: authenticate success, global_id 4599 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 10 monclient: _renew_subs 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e0140e60 con 0x7f93e014be90 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 10 monclient: _renew_subs 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e0142c40 con 0x7f93e014be90 2024-03-25T19:28:34.721 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.720+0000 7f93e7493640 1 librados: init done 2024-03-25T19:28:34.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.721+0000 7f93e7493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:34.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.721+0000 7f93e7493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:34.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.721+0000 7f93e7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:34.722 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:34.721+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e00627b0 con 0x7f93e014be90 2024-03-25T19:28:35.036 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93d8003c20 con 0x7f93e014be90 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f93d8004030 con 0x7f93e014be90 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d5ffb640 10 monclient: dump: 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:35.037 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:35.038 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:35.038 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:35.038 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:35.038 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:35.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:28:35.038 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.037+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f93d8004250 con 0x7f93e014be90 2024-03-25T19:28:35.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.038+0000 7f93e522f640 10 monclient: get_auth_request con 0x7f93c406dd00 auth_method 0 2024-03-25T19:28:35.039 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.038+0000 7f93d5ffb640 10 client.4599.objecter ms_handle_connect 0x7f93c406dd00 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f93d8054b50 con 0x7f93e014be90 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 10 client.4599.objecter ms_dispatch 0x7f93e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 3 client.4599.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 3 client.4599.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93d8050660 con 0x7f93e014be90 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:35.046 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.046+0000 7f93d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:35.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.295+0000 7f93e7493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:35.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.295+0000 7f93e7493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:35.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.295+0000 7f93e7493640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:28:35.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.295+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93e00b5cf0 con 0x7f93e014be90 2024-03-25T19:28:35.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.318+0000 7f93d5ffb640 1 -- 172.21.15.60:0/2594153590 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f93d8052590 con 0x7f93e014be90 2024-03-25T19:28:35.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.318+0000 7f93d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:35.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.318+0000 7f93d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:35.318 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:35.318 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:35.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.320+0000 7f93e7493640 10 librados: watch_flush enter 2024-03-25T19:28:35.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.320+0000 7f93e7493640 10 librados: watch_flush exit 2024-03-25T19:28:35.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.320+0000 7f93e7493640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.320+0000 7f93e7493640 10 client.4599.objecter successfully canceled tick 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.320+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 >> v1:172.21.15.90:6800/1827638754 conn(0x7f93c406dd00 legacy=0x7f93c40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 10 monclient: shutdown 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 >> v1:172.21.15.60:6789/0 conn(0x7f93e014be90 legacy=0x7f93e01e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 shutdown_connections 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 >> 172.21.15.60:0/2594153590 conn(0x7f93e00617a0 msgr2=0x7f93e00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 shutdown_connections 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 1 -- 172.21.15.60:0/2594153590 wait complete. 2024-03-25T19:28:35.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:35.321+0000 7f93e7493640 1 librados: shutdown 2024-03-25T19:28:35.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:40.480 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:41.191 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.534 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:41.245 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.580 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:08:41.292 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.585 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:41.239 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.599 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:41.253 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.604 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:41.315 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.608 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:41.319 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.627 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:08:41.339 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:40.630 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:08:41.284 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:28:42.136 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:42.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.252+0000 7f75722e6640 10 monclient: get_monmap_and_config 2024-03-25T19:28:42.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.252+0000 7f75722e6640 10 monclient: build_initial_monmap 2024-03-25T19:28:42.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.252+0000 7f75722e6640 10 monclient: monmap: 2024-03-25T19:28:42.251 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:42.252785+0000 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:42.252785+0000 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:42.252 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:42.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.253+0000 7f75722e6640 1 Processor -- start 2024-03-25T19:28:42.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.253+0000 7f75722e6640 1 -- start start 2024-03-25T19:28:42.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.253+0000 7f75722e6640 10 monclient: init 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 10 monclient(hunting): picked mon.noname-b con 0x7f756c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 10 monclient(hunting): picked mon.noname-c con 0x7f756c0b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 10 monclient(hunting): picked mon.noname-a con 0x7f756c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c0af940 con 0x7f756c14be90 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c0b0a70 con 0x7f756c0b1730 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c0b0d50 con 0x7f756c0b4d60 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:42.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.254+0000 7f75722e6640 10 monclient(hunting): authenticate will time out at 6849.352374s 2024-03-25T19:28:42.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.270+0000 7f756bfff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f756c0b4d60 0x7f756c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:44038/0 (socket says 172.21.15.60:44038) 2024-03-25T19:28:42.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.270+0000 7f756bfff640 1 -- 172.21.15.60:0/2595568258 learned_addr learned my addr 172.21.15.60:0/2595568258 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:42.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.280+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349644677 0 0) 0x7f756c0b0a70 con 0x7f756c0b1730 2024-03-25T19:28:42.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.280+0000 7f756affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:42.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.280+0000 7f756affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:42.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.280+0000 7f756affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:42.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.280+0000 7f756affd640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:28:42.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.280+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7554003520 con 0x7f756c0b1730 2024-03-25T19:28:42.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.286+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918861971 0 0) 0x7f756c0b0d50 con 0x7f756c0b4d60 2024-03-25T19:28:42.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.286+0000 7f756affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:42.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.286+0000 7f756affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:42.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.286+0000 7f756affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:42.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.286+0000 7f756affd640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:28:42.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.286+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c0b0a70 con 0x7f756c0b4d60 2024-03-25T19:28:42.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.291+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860748792 0 0) 0x7f756c0af940 con 0x7f756c14be90 2024-03-25T19:28:42.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.291+0000 7f756affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:42.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.291+0000 7f756affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:42.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.291+0000 7f756affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:42.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.291+0000 7f756affd640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:28:42.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.291+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c0b0d50 con 0x7f756c14be90 2024-03-25T19:28:42.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700411148 0 0) 0x7f756c0b0a70 con 0x7f756c0b4d60 2024-03-25T19:28:42.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 >> v1:172.21.15.90:6789/0 conn(0x7f756c0b1730 legacy=0x7f756c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:42.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 >> v1:172.21.15.60:6789/0 conn(0x7f756c14be90 legacy=0x7f756c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 1 monclient: found mon.noname-b 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c0b1030 con 0x7f756c0b4d60 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f755c001670 con 0x7f756c0b4d60 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f756affd640 10 monclient: dump: 2024-03-25T19:28:42.302 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f75722e6640 5 monclient: authenticate success, global_id 4589 2024-03-25T19:28:42.303 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.301+0000 7f75722e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f755c001890 con 0x7f756c0b4d60 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f756affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f756affd640 1 -- 172.21.15.60:0/2595568258 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f755c001c20 con 0x7f756c0b4d60 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f756affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f756affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f756affd640 10 monclient: dump: 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:42.310 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:42.311 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:42.311 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:42.311 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:42.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f75722e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:42.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.310+0000 7f75722e6640 4 set_mon_vals no callback set 2024-03-25T19:28:42.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.311+0000 7f75722e6640 10 monclient: shutdown 2024-03-25T19:28:42.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.311+0000 7f75722e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:42.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.311+0000 7f75722e6640 1 -- 172.21.15.60:0/2595568258 >> v1:172.21.15.60:6790/0 conn(0x7f756c0b4d60 legacy=0x7f756c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:42.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.311+0000 7f75722e6640 1 -- 172.21.15.60:0/2595568258 shutdown_connections 2024-03-25T19:28:42.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.311+0000 7f75722e6640 1 -- 172.21.15.60:0/2595568258 >> 172.21.15.60:0/2595568258 conn(0x7f756c0617a0 msgr2=0x7f756c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:42.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.311+0000 7f75722e6640 1 -- 172.21.15.60:0/2595568258 shutdown_connections 2024-03-25T19:28:42.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.311+0000 7f75722e6640 1 -- 172.21.15.60:0/2595568258 wait complete. 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 10 monclient: build_initial_monmap 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 10 monclient: monmap: 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 1 librados: starting msgr at 2024-03-25T19:28:42.314 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 1 librados: starting objecter 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 1 Processor -- start 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 1 -- start start 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 1 librados: setting wanted keys 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 1 librados: calling monclient init 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.312+0000 7f75722e6640 10 monclient: init 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 10 monclient(hunting): picked mon.noname-c con 0x7f756c0b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 10 monclient(hunting): picked mon.noname-b con 0x7f756c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 10 monclient(hunting): picked mon.noname-a con 0x7f756c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c1db0d0 con 0x7f756c14be90 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c1dbd00 con 0x7f756c0b4d60 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756c1dbe90 con 0x7f756c0b1730 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:42.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.313+0000 7f75722e6640 10 monclient(hunting): authenticate will time out at 6849.411213s 2024-03-25T19:28:42.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.315+0000 7f756bfff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f756c0b1730 0x7f756c1db5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:44084/0 (socket says 172.21.15.60:44084) 2024-03-25T19:28:42.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.315+0000 7f756bfff640 1 -- 172.21.15.60:0/760382062 learned_addr learned my addr 172.21.15.60:0/760382062 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:42.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.316+0000 7f7568ff9640 10 client.?.objecter ms_handle_connect 0x7f756c0b1730 2024-03-25T19:28:42.316 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.316+0000 7f7568ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:42.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.325+0000 7f7568ff9640 10 client.?.objecter ms_handle_connect 0x7f756c14be90 2024-03-25T19:28:42.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.325+0000 7f7568ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:42.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.326+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855119751 0 0) 0x7f756c0b1030 con 0x7f756c0b1730 2024-03-25T19:28:42.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.326+0000 7f7568ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:42.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.326+0000 7f7568ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:42.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.326+0000 7f7568ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:42.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.326+0000 7f7568ff9640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:28:42.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.326+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7558004eb0 con 0x7f756c0b1730 2024-03-25T19:28:42.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.330+0000 7f7568ff9640 10 client.?.objecter ms_handle_connect 0x7f756c0b4d60 2024-03-25T19:28:42.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.330+0000 7f7568ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:42.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.332+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107801236 0 0) 0x7f756c1db0d0 con 0x7f756c14be90 2024-03-25T19:28:42.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.332+0000 7f7568ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:28:42.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.332+0000 7f7568ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:42.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.332+0000 7f7568ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:42.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.332+0000 7f7568ff9640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:28:42.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.332+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756c0b1030 con 0x7f756c14be90 2024-03-25T19:28:42.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.342+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4193895067 0 0) 0x7f7558004eb0 con 0x7f756c0b1730 2024-03-25T19:28:42.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.342+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f756c1db0d0 con 0x7f756c0b1730 2024-03-25T19:28:42.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.342+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f755c001510 con 0x7f756c0b1730 2024-03-25T19:28:42.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.342+0000 7f7568ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.342+0000 7f7568ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.342+0000 7f7568ff9640 10 monclient(hunting): dump: 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:42.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.342+0000 7f7568ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3298017307 0 0) 0x7f756c1db0d0 con 0x7f756c0b1730 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 >> v1:172.21.15.90:6789/0 conn(0x7f756c0b4d60 legacy=0x7f756c13b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 >> v1:172.21.15.60:6789/0 conn(0x7f756c14be90 legacy=0x7f756c1e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 1 monclient: found mon.c 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f756c13be30 con 0x7f756c0b1730 2024-03-25T19:28:42.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 10 monclient: _finish_auth 0 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f7568ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 5 monclient: authenticate success, global_id 4592 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 10 monclient: _renew_subs 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756c13c080 con 0x7f756c0b1730 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 10 monclient: _renew_subs 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f756c1399b0 con 0x7f756c0b1730 2024-03-25T19:28:42.348 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.347+0000 7f75722e6640 1 librados: init done 2024-03-25T19:28:42.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.348+0000 7f75722e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:42.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.348+0000 7f75722e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:42.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.348+0000 7f75722e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:42.349 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.348+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f756c1401f0 con 0x7f756c0b1730 2024-03-25T19:28:42.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.666+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f755c004a90 con 0x7f756c0b1730 2024-03-25T19:28:42.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.666+0000 7f7568ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:42.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.666+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f755c004da0 con 0x7f756c0b1730 2024-03-25T19:28:42.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.666+0000 7f7568ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.666+0000 7f7568ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.666+0000 7f7568ff9640 10 monclient: dump: 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:42.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.666+0000 7f756a7fc640 4 set_mon_vals no callback set 2024-03-25T19:28:42.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.667+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f755c01ad70 con 0x7f756c0b1730 2024-03-25T19:28:42.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.668+0000 7f756b7fe640 10 monclient: get_auth_request con 0x7f755806dd40 auth_method 0 2024-03-25T19:28:42.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.668+0000 7f7568ff9640 10 client.4592.objecter ms_handle_connect 0x7f755806dd40 2024-03-25T19:28:42.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.674+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f755c054870 con 0x7f756c0b1730 2024-03-25T19:28:42.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.675+0000 7f7568ff9640 10 client.4592.objecter ms_dispatch 0x7f756c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:42.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.675+0000 7f7568ff9640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:42.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.675+0000 7f7568ff9640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:42.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.675+0000 7f7568ff9640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T19:28:42.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.675+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f755c050380 con 0x7f756c0b1730 2024-03-25T19:28:42.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.675+0000 7f7568ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:42.675 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.675+0000 7f7568ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:42.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.920+0000 7f75722e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:42.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.920+0000 7f75722e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:42.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.920+0000 7f75722e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:42.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.920+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f756c0627b0 con 0x7f756c0b1730 2024-03-25T19:28:42.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.943+0000 7f7568ff9640 1 -- 172.21.15.60:0/760382062 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f755c0522b0 con 0x7f756c0b1730 2024-03-25T19:28:42.942 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.943+0000 7f7568ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:42.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.943+0000 7f7568ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:42.943 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:42.943 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:42.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 10 librados: watch_flush enter 2024-03-25T19:28:42.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 10 librados: watch_flush exit 2024-03-25T19:28:42.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T19:28:42.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 10 client.4592.objecter successfully canceled tick 2024-03-25T19:28:42.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 >> v1:172.21.15.90:6800/1827638754 conn(0x7f755806dd40 legacy=0x7f7558070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:42.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 10 monclient: shutdown 2024-03-25T19:28:42.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:42.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 >> v1:172.21.15.60:6790/0 conn(0x7f756c0b1730 legacy=0x7f756c1db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:42.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 shutdown_connections 2024-03-25T19:28:42.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 >> 172.21.15.60:0/760382062 conn(0x7f756c0617a0 msgr2=0x7f756c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:42.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 shutdown_connections 2024-03-25T19:28:42.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.945+0000 7f75722e6640 1 -- 172.21.15.60:0/760382062 wait complete. 2024-03-25T19:28:42.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:42.946+0000 7f75722e6640 1 librados: shutdown 2024-03-25T19:28:42.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:49.764 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:49.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.882+0000 7f9ea0a86640 10 monclient: get_monmap_and_config 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.882+0000 7f9ea0a86640 10 monclient: build_initial_monmap 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.882+0000 7f9ea0a86640 10 monclient: monmap: 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:49.883017+0000 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:49.883017+0000 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:49.882 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.882+0000 7f9ea0a86640 1 Processor -- start 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.883+0000 7f9ea0a86640 1 -- start start 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.883+0000 7f9ea0a86640 10 monclient: init 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.883+0000 7f9ea0a86640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.883+0000 7f9ea0a86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.883+0000 7f9ea0a86640 10 monclient(hunting): picked mon.noname-a con 0x7f9e9c0b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.883+0000 7f9ea0a86640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9c0b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.884+0000 7f9ea0a86640 10 monclient(hunting): picked mon.noname-b con 0x7f9e9c14be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.884+0000 7f9ea0a86640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c0af940 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.884+0000 7f9ea0a86640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c0b0a70 con 0x7f9e9c0b1730 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.884+0000 7f9ea0a86640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c0b0d50 con 0x7f9e9c14be90 2024-03-25T19:28:49.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.884+0000 7f9ea0a86640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:49.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.884+0000 7f9ea0a86640 10 monclient(hunting): authenticate will time out at 6856.981932s 2024-03-25T19:28:49.887 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.888+0000 7f9e99d9b640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9e9c0b1730 0x7f9e9c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:37214/0 (socket says 172.21.15.60:37214) 2024-03-25T19:28:49.887 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.888+0000 7f9e99d9b640 1 -- 172.21.15.60:0/1043995966 learned_addr learned my addr 172.21.15.60:0/1043995966 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:49.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.906+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516371168 0 0) 0x7f9e9c0b0a70 con 0x7f9e9c0b1730 2024-03-25T19:28:49.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.906+0000 7f9e9959a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:49.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.906+0000 7f9e9959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:49.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.906+0000 7f9e9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:49.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.906+0000 7f9e9959a640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:28:49.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.906+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e84003520 con 0x7f9e9c0b1730 2024-03-25T19:28:49.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.920+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765091946 0 0) 0x7f9e9c0b0d50 con 0x7f9e9c14be90 2024-03-25T19:28:49.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.920+0000 7f9e9959a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:49.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.920+0000 7f9e9959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:49.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.920+0000 7f9e9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:49.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.921+0000 7f9e9959a640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:28:49.920 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.921+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9c0b0a70 con 0x7f9e9c14be90 2024-03-25T19:28:49.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.926+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746712521 0 0) 0x7f9e9c0af940 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.926+0000 7f9e9959a640 10 monclient(hunting): _negotiate 2024-03-25T19:28:49.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.926+0000 7f9e9959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:49.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.926+0000 7f9e9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:49.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.926+0000 7f9e9959a640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:28:49.926 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.926+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9c0b0d50 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 625470823 0 0) 0x7f9e84003520 con 0x7f9e9c0b1730 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 >> v1:172.21.15.60:6790/0 conn(0x7f9e9c14be90 legacy=0x7f9e9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 >> v1:172.21.15.60:6789/0 conn(0x7f9e9c0b4d60 legacy=0x7f9e9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 10 monclient: _finish_hunting 0 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 1 monclient: found mon.noname-c 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:28:49.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c0b1030 con 0x7f9e9c0b1730 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 10 monclient: _finish_auth 0 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 10 monclient: _check_auth_tickets 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9e90001690 con 0x7f9e9c0b1730 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9e9959a640 10 monclient: dump: 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:49.933 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:49.934 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:49.934 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:49.934 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:49.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9ea0a86640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:28:49.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.932+0000 7f9ea0a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:49.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e900017f0 con 0x7f9e9c0b1730 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9e9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9e9959a640 1 -- 172.21.15.60:0/1043995966 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9e90001c00 con 0x7f9e9c0b1730 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9e9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9e9959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9e9959a640 10 monclient: dump: 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:49.950 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:49.951 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:49.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9ea0a86640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:49.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.950+0000 7f9ea0a86640 4 set_mon_vals no callback set 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.951+0000 7f9ea0a86640 10 monclient: shutdown 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.951+0000 7f9ea0a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.951+0000 7f9ea0a86640 1 -- 172.21.15.60:0/1043995966 >> v1:172.21.15.90:6789/0 conn(0x7f9e9c0b1730 legacy=0x7f9e9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.951+0000 7f9ea0a86640 1 -- 172.21.15.60:0/1043995966 shutdown_connections 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.951+0000 7f9ea0a86640 1 -- 172.21.15.60:0/1043995966 >> 172.21.15.60:0/1043995966 conn(0x7f9e9c0617a0 msgr2=0x7f9e9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.951+0000 7f9ea0a86640 1 -- 172.21.15.60:0/1043995966 shutdown_connections 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.951+0000 7f9ea0a86640 1 -- 172.21.15.60:0/1043995966 wait complete. 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.952+0000 7f9ea0a86640 10 monclient: build_initial_monmap 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.952+0000 7f9ea0a86640 10 monclient: monmap: 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:49.952 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:49.953 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:49.953 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:49.953 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:49.953 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:49.953 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:49.953 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:49.953 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.953+0000 7f9ea0a86640 1 librados: starting msgr at 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.953+0000 7f9ea0a86640 1 librados: starting objecter 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.953+0000 7f9ea0a86640 1 Processor -- start 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.953+0000 7f9ea0a86640 1 -- start start 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.953+0000 7f9ea0a86640 1 librados: setting wanted keys 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.953+0000 7f9ea0a86640 1 librados: calling monclient init 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.953+0000 7f9ea0a86640 10 monclient: init 2024-03-25T19:28:49.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 10 monclient(hunting): picked mon.noname-b con 0x7f9e9c0b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 10 monclient(hunting): picked mon.noname-c con 0x7f9e9c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 10 monclient(hunting): picked mon.noname-a con 0x7f9e9c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c13a900 con 0x7f9e9c14be90 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c13ab70 con 0x7f9e9c0b1730 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e9c13ae50 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:49.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.954+0000 7f9ea0a86640 10 monclient(hunting): authenticate will time out at 6857.052063s 2024-03-25T19:28:49.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.963+0000 7f9e99d9b640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f9e9c0b4d60 0x7f9e9c140d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:57378/0 (socket says 172.21.15.60:57378) 2024-03-25T19:28:49.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.963+0000 7f9e99d9b640 1 -- 172.21.15.60:0/4231633068 learned_addr learned my addr 172.21.15.60:0/4231633068 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:49.963 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.963+0000 7f9e8affd640 10 client.?.objecter ms_handle_connect 0x7f9e9c0b4d60 2024-03-25T19:28:49.963 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.963+0000 7f9e8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:49.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.966+0000 7f9e8affd640 10 client.?.objecter ms_handle_connect 0x7f9e9c0b1730 2024-03-25T19:28:49.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.966+0000 7f9e8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:49.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.968+0000 7f9e8affd640 10 client.?.objecter ms_handle_connect 0x7f9e9c14be90 2024-03-25T19:28:49.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.968+0000 7f9e8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:49.970 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686602571 0 0) 0x7f9e9c13ab70 con 0x7f9e9c0b1730 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e7c004eb0 con 0x7f9e9c0b1730 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112235096 0 0) 0x7f9e9c13ae50 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:28:49.971 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.971+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9c13ab70 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.980+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507578262 0 0) 0x7f9e9c13a900 con 0x7f9e9c14be90 2024-03-25T19:28:49.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.980+0000 7f9e8affd640 10 monclient(hunting): _negotiate 2024-03-25T19:28:49.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.980+0000 7f9e8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:49.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.980+0000 7f9e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:49.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.980+0000 7f9e8affd640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:28:49.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.980+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9c13ae50 con 0x7f9e9c14be90 2024-03-25T19:28:49.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.986+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1286556179 0 0) 0x7f9e9c13ab70 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.986+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e9c13a900 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.986+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9e90001b40 con 0x7f9e9c0b4d60 2024-03-25T19:28:49.985 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.986+0000 7f9e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.986+0000 7f9e8affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.986+0000 7f9e8affd640 10 monclient(hunting): dump: 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:49.986 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:49.987 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.986+0000 7f9e8affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.994+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636174989 0 0) 0x7f9e7c004eb0 con 0x7f9e9c0b1730 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.994+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e9c13ab70 con 0x7f9e9c0b1730 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.994+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9e8c0018f0 con 0x7f9e9c0b1730 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.994+0000 7f9e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.994+0000 7f9e8affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.994+0000 7f9e8affd640 10 monclient(hunting): dump: 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.994 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:49.995 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:49.995 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:49.995 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:49.995 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:49.995 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:49.995 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2153142372 0 0) 0x7f9e9c13ae50 con 0x7f9e9c14be90 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e7c004eb0 con 0x7f9e9c14be90 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9e94001bd0 con 0x7f9e9c14be90 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 10 monclient(hunting): dump: 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:50.001 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3361155750 0 0) 0x7f9e9c13a900 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 >> v1:172.21.15.90:6789/0 conn(0x7f9e9c0b1730 legacy=0x7f9e9c139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 >> v1:172.21.15.60:6789/0 conn(0x7f9e9c14be90 legacy=0x7f9e9c141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 10 monclient: _finish_hunting 0 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 monclient: found mon.c 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:50.002 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c140840 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 10 monclient: _finish_auth 0 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 10 monclient: _check_auth_tickets 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9ea0a86640 5 monclient: authenticate success, global_id 4598 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9ea0a86640 10 monclient: _renew_subs 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9ea0a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:49.999+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9c140510 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.000+0000 7f9ea0a86640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.000+0000 7f9ea0a86640 10 monclient: _renew_subs 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.000+0000 7f9ea0a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.000+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9c1e7a10 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.000+0000 7f9ea0a86640 1 librados: init done 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.001+0000 7f9ea0a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:50.003 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.001+0000 7f9ea0a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:50.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.001+0000 7f9ea0a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:50.004 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.001+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9c0afbd0 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.320+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e900011f0 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.320+0000 7f9e8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:50.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.320+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9e90004c30 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.320+0000 7f9e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:50.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.320+0000 7f9e8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.320+0000 7f9e8affd640 10 monclient: dump: 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:50.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.320+0000 7f9e98d99640 4 set_mon_vals no callback set 2024-03-25T19:28:50.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.321+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f9e9001ac40 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.322+0000 7f9e9a59c640 10 monclient: get_auth_request con 0x7f9e7c06e060 auth_method 0 2024-03-25T19:28:50.322 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.322+0000 7f9e8affd640 10 client.4598.objecter ms_handle_connect 0x7f9e7c06e060 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f9e90054890 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 10 client.4598.objecter ms_dispatch 0x7f9e9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e900501d0 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:50.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.328+0000 7f9e8affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:50.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.585+0000 7f9ea0a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:50.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.585+0000 7f9ea0a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:50.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.585+0000 7f9ea0a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:28:50.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.585+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e9c1409d0 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.606+0000 7f9e8affd640 1 -- 172.21.15.60:0/4231633068 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f9e900522d0 con 0x7f9e9c0b4d60 2024-03-25T19:28:50.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.606+0000 7f9e8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:50.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.606+0000 7f9e8affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:50.606 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:50.606 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.608+0000 7f9ea0a86640 10 librados: watch_flush enter 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.608+0000 7f9ea0a86640 10 librados: watch_flush exit 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.608+0000 7f9ea0a86640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.608+0000 7f9ea0a86640 10 client.4598.objecter successfully canceled tick 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 >> v1:172.21.15.90:6800/1827638754 conn(0x7f9e7c06e060 legacy=0x7f9e7c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 10 monclient: shutdown 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 >> v1:172.21.15.60:6790/0 conn(0x7f9e9c0b4d60 legacy=0x7f9e9c140d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 shutdown_connections 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 >> 172.21.15.60:0/4231633068 conn(0x7f9e9c0617a0 msgr2=0x7f9e9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 shutdown_connections 2024-03-25T19:28:50.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 1 -- 172.21.15.60:0/4231633068 wait complete. 2024-03-25T19:28:50.610 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:50.609+0000 7f9ea0a86640 1 librados: shutdown 2024-03-25T19:28:50.620 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:28:57.427 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:28:57.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.543+0000 7f2e1aaa1640 10 monclient: get_monmap_and_config 2024-03-25T19:28:57.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.543+0000 7f2e1aaa1640 10 monclient: build_initial_monmap 2024-03-25T19:28:57.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.543+0000 7f2e1aaa1640 10 monclient: monmap: 2024-03-25T19:28:57.543 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:57.543 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:28:57.543761+0000 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:28:57.543761+0000 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:28:57.544 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.544+0000 7f2e1aaa1640 1 Processor -- start 2024-03-25T19:28:57.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.544+0000 7f2e1aaa1640 1 -- start start 2024-03-25T19:28:57.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.544+0000 7f2e1aaa1640 10 monclient: init 2024-03-25T19:28:57.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.544+0000 7f2e1aaa1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:57.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.544+0000 7f2e1aaa1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:28:57.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.544+0000 7f2e1aaa1640 10 monclient(hunting): picked mon.noname-a con 0x7f2e140b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:57.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.544+0000 7f2e1aaa1640 10 monclient(hunting): picked mon.noname-b con 0x7f2e140b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:57.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.545+0000 7f2e1aaa1640 10 monclient(hunting): picked mon.noname-c con 0x7f2e1414be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:57.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.545+0000 7f2e1aaa1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e140af940 con 0x7f2e140b4d60 2024-03-25T19:28:57.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.545+0000 7f2e1aaa1640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e140b0a70 con 0x7f2e1414be90 2024-03-25T19:28:57.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.545+0000 7f2e1aaa1640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e140b0d50 con 0x7f2e140b1730 2024-03-25T19:28:57.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.545+0000 7f2e1aaa1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:57.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.545+0000 7f2e1aaa1640 10 monclient(hunting): authenticate will time out at 6864.643033s 2024-03-25T19:28:57.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.550+0000 7f2e1903e640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2e1414be90 0x7f2e1414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:47838/0 (socket says 172.21.15.60:47838) 2024-03-25T19:28:57.550 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.550+0000 7f2e1903e640 1 -- 172.21.15.60:0/2815106272 learned_addr learned my addr 172.21.15.60:0/2815106272 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:57.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.566+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744170526 0 0) 0x7f2e140b0d50 con 0x7f2e140b1730 2024-03-25T19:28:57.565 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.566+0000 7f2e137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.566+0000 7f2e137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.566+0000 7f2e137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.566+0000 7f2e137fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:28:57.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.566+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e04003520 con 0x7f2e140b1730 2024-03-25T19:28:57.574 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.575+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372144305 0 0) 0x7f2e140b0a70 con 0x7f2e1414be90 2024-03-25T19:28:57.575 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.575+0000 7f2e137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.575 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.575+0000 7f2e137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.575 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.575+0000 7f2e137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.575 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.575+0000 7f2e137fe640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:28:57.575 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.575+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e140b0d50 con 0x7f2e1414be90 2024-03-25T19:28:57.587 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1143269909 0 0) 0x7f2e04003520 con 0x7f2e140b1730 2024-03-25T19:28:57.587 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 >> v1:172.21.15.90:6789/0 conn(0x7f2e1414be90 legacy=0x7f2e1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.587 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 >> v1:172.21.15.60:6789/0 conn(0x7f2e140b4d60 legacy=0x7f2e1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.587 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:28:57.587 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 1 monclient: found mon.noname-b 2024-03-25T19:28:57.587 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:57.587 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140b1030 con 0x7f2e140b1730 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 10 monclient: _finish_auth 0 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e00001690 con 0x7f2e140b1730 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e137fe640 10 monclient: dump: 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:57.588 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:57.589 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:57.589 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.589 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e1aaa1640 5 monclient: authenticate success, global_id 4601 2024-03-25T19:28:57.589 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.587+0000 7f2e1aaa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:28:57.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e000017f0 con 0x7f2e140b1730 2024-03-25T19:28:57.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:57.599 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e137fe640 1 -- 172.21.15.60:0/2815106272 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e00001c00 con 0x7f2e140b1730 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e1aaa1640 10 monclient: get_monmap_and_config success 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e1aaa1640 4 set_mon_vals no callback set 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e137fe640 10 monclient: dump: 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:57.600 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:57.601 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:57.601 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:57.601 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e1aaa1640 10 monclient: shutdown 2024-03-25T19:28:57.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e1aaa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:57.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.600+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2815106272 >> v1:172.21.15.60:6790/0 conn(0x7f2e140b1730 legacy=0x7f2e140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.601+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2815106272 shutdown_connections 2024-03-25T19:28:57.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.601+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2815106272 >> 172.21.15.60:0/2815106272 conn(0x7f2e140617a0 msgr2=0x7f2e14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:57.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.601+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2815106272 shutdown_connections 2024-03-25T19:28:57.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.601+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2815106272 wait complete. 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.601+0000 7f2e1aaa1640 10 monclient: build_initial_monmap 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.601+0000 7f2e1aaa1640 10 monclient: monmap: 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.604 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.602+0000 7f2e1aaa1640 1 librados: starting msgr at 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.602+0000 7f2e1aaa1640 1 librados: starting objecter 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.602+0000 7f2e1aaa1640 1 Processor -- start 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.602+0000 7f2e1aaa1640 1 -- start start 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.602+0000 7f2e1aaa1640 1 librados: setting wanted keys 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.602+0000 7f2e1aaa1640 1 librados: calling monclient init 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.602+0000 7f2e1aaa1640 10 monclient: init 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 10 monclient: _reopen_session rank -1 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 10 monclient(hunting): picked mon.noname-b con 0x7f2e140b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 10 monclient(hunting): picked mon.noname-c con 0x7f2e140b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:28:57.605 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 10 monclient(hunting): picked mon.noname-a con 0x7f2e1414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:28:57.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e14137e70 con 0x7f2e1414be90 2024-03-25T19:28:57.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e141380e0 con 0x7f2e140b1730 2024-03-25T19:28:57.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e14135ff0 con 0x7f2e140b4d60 2024-03-25T19:28:57.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 10 monclient(hunting): _renew_subs 2024-03-25T19:28:57.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.603+0000 7f2e1aaa1640 10 monclient(hunting): authenticate will time out at 6864.701185s 2024-03-25T19:28:57.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.604+0000 7f2e1883d640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2e140b1730 0x7f2e1412fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:47894/0 (socket says 172.21.15.60:47894) 2024-03-25T19:28:57.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.604+0000 7f2e1883d640 1 -- 172.21.15.60:0/2960481435 learned_addr learned my addr 172.21.15.60:0/2960481435 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:28:57.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.605+0000 7f2e117fa640 10 client.?.objecter ms_handle_connect 0x7f2e140b1730 2024-03-25T19:28:57.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.605+0000 7f2e117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:57.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.666+0000 7f2e117fa640 10 client.?.objecter ms_handle_connect 0x7f2e140b4d60 2024-03-25T19:28:57.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.667+0000 7f2e117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:57.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.667+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453622964 0 0) 0x7f2e141380e0 con 0x7f2e140b1730 2024-03-25T19:28:57.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.667+0000 7f2e117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.667+0000 7f2e117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.667+0000 7f2e117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.667+0000 7f2e117fa640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:28:57.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.668+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df8004eb0 con 0x7f2e140b1730 2024-03-25T19:28:57.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.668+0000 7f2e117fa640 10 client.?.objecter ms_handle_connect 0x7f2e1414be90 2024-03-25T19:28:57.668 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.668+0000 7f2e117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:28:57.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.679+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1388568206 0 0) 0x7f2df8004eb0 con 0x7f2e140b1730 2024-03-25T19:28:57.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.679+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e141380e0 con 0x7f2e140b1730 2024-03-25T19:28:57.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.679+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e080019d0 con 0x7f2e140b1730 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.679+0000 7f2e117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.679+0000 7f2e117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.679+0000 7f2e117fa640 10 monclient(hunting): dump: 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.680 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.679+0000 7f2e117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.680+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264834779 0 0) 0x7f2e14137e70 con 0x7f2e1414be90 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.680+0000 7f2e117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.680+0000 7f2e117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.680+0000 7f2e117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.680+0000 7f2e117fa640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.680+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df8004eb0 con 0x7f2e1414be90 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.682+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831213826 0 0) 0x7f2e14135ff0 con 0x7f2e140b4d60 2024-03-25T19:28:57.681 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.682+0000 7f2e117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:28:57.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.682+0000 7f2e117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:28:57.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.682+0000 7f2e117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:28:57.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.682+0000 7f2e117fa640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:28:57.682 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.682+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e14137e70 con 0x7f2e140b4d60 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.693+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3950954745 0 0) 0x7f2e14137e70 con 0x7f2e140b4d60 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.693+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e14135ff0 con 0x7f2e140b4d60 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.693+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e00001b00 con 0x7f2e140b4d60 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.693+0000 7f2e117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.693+0000 7f2e117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.693+0000 7f2e117fa640 10 monclient(hunting): dump: 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.693 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.694 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:57.694 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:57.694 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:57.694 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:57.694 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.705+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 637252034 0 0) 0x7f2df8004eb0 con 0x7f2e1414be90 2024-03-25T19:28:57.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.705+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e14137e70 con 0x7f2e1414be90 2024-03-25T19:28:57.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.705+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e0c001bd0 con 0x7f2e1414be90 2024-03-25T19:28:57.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.705+0000 7f2e117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.705+0000 7f2e117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:28:57.705 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.705+0000 7f2e117fa640 10 monclient(hunting): dump: 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:57.706 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2624466113 0 0) 0x7f2e141380e0 con 0x7f2e140b1730 2024-03-25T19:28:57.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 >> v1:172.21.15.60:6790/0 conn(0x7f2e140b4d60 legacy=0x7f2e141d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 >> v1:172.21.15.60:6789/0 conn(0x7f2e1414be90 legacy=0x7f2e141e3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:57.711 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 10 monclient: _finish_hunting 0 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 1 monclient: found mon.b 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14135c10 con 0x7f2e140b1730 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 10 monclient: _finish_auth 0 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 10 monclient: _check_auth_tickets 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 5 monclient: authenticate success, global_id 4606 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 10 monclient: _renew_subs 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14130590 con 0x7f2e140b1730 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 10 monclient: _renew_subs 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:57.712 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e141367c0 con 0x7f2e140b1730 2024-03-25T19:28:57.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.711+0000 7f2e1aaa1640 1 librados: init done 2024-03-25T19:28:57.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.712+0000 7f2e1aaa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.712+0000 7f2e1aaa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.712+0000 7f2e1aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:57.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.712+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14135e00 con 0x7f2e140b1730 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.993+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e08001c50 con 0x7f2e140b1730 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.993+0000 7f2e117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.993+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e08003a60 con 0x7f2e140b1730 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.993+0000 7f2e117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.993+0000 7f2e117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.993+0000 7f2e117fa640 10 monclient: dump: 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.993 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:28:57.994 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:28:57.994 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:28:57.994 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:28:57.994 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:28:57.994 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:28:57.994 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:28:57.994 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.993+0000 7f2e12ffd640 4 set_mon_vals no callback set 2024-03-25T19:28:57.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.994+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2e08003c80 con 0x7f2e140b1730 2024-03-25T19:28:57.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.995+0000 7f2e13fff640 10 monclient: get_auth_request con 0x7f2df806e1a0 auth_method 0 2024-03-25T19:28:57.995 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.995+0000 7f2e117fa640 10 client.4606.objecter ms_handle_connect 0x7f2df806e1a0 2024-03-25T19:28:57.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.998+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f2e08054760 con 0x7f2e140b1730 2024-03-25T19:28:57.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.998+0000 7f2e117fa640 10 client.4606.objecter ms_dispatch 0x7f2e14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:28:57.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.998+0000 7f2e117fa640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:28:57.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.998+0000 7f2e117fa640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:28:57.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.999+0000 7f2e117fa640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T19:28:57.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.999+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e08050270 con 0x7f2e140b1730 2024-03-25T19:28:57.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.999+0000 7f2e117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:28:57.999 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:57.999+0000 7f2e117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:28:58.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.247+0000 7f2e1aaa1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:58.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.247+0000 7f2e1aaa1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:58.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.247+0000 7f2e1aaa1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:28:58.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.247+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e140627b0 con 0x7f2e140b1730 2024-03-25T19:28:58.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.268+0000 7f2e117fa640 1 -- 172.21.15.60:0/2960481435 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f2e080521a0 con 0x7f2e140b1730 2024-03-25T19:28:58.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.269+0000 7f2e117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:28:58.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.269+0000 7f2e117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:28:58.268 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:28:58.269 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:28:58.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 10 librados: watch_flush enter 2024-03-25T19:28:58.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 10 librados: watch_flush exit 2024-03-25T19:28:58.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 10 client.4606.objecter successfully canceled tick 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 >> v1:172.21.15.90:6800/1827638754 conn(0x7f2df806e1a0 legacy=0x7f2df8070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 10 monclient: shutdown 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 >> v1:172.21.15.90:6789/0 conn(0x7f2e140b1730 legacy=0x7f2e1412fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 shutdown_connections 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 >> 172.21.15.60:0/2960481435 conn(0x7f2e140617a0 msgr2=0x7f2e1413c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 shutdown_connections 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 1 -- 172.21.15.60:0/2960481435 wait complete. 2024-03-25T19:28:58.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:28:58.271+0000 7f2e1aaa1640 1 librados: shutdown 2024-03-25T19:28:58.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:00.560 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:01.271 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.616 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:01.327 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.655 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:01.366 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.663 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:01.374 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.664 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:01.318 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.669 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:01.323 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.671 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:01.382 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.702 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:01.356 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:00.705 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:01.416 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:05.088 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:05.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.231+0000 7f03f86ba640 10 monclient: get_monmap_and_config 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.231+0000 7f03f86ba640 10 monclient: build_initial_monmap 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.231+0000 7f03f86ba640 10 monclient: monmap: 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:05.232079+0000 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:05.232079+0000 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:05.231 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.232+0000 7f03f86ba640 1 Processor -- start 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.232+0000 7f03f86ba640 1 -- start start 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.232+0000 7f03f86ba640 10 monclient: init 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 10 monclient(hunting): picked mon.noname-c con 0x7f03f00b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 10 monclient(hunting): picked mon.noname-a con 0x7f03f00b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 10 monclient(hunting): picked mon.noname-b con 0x7f03f014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f00af940 con 0x7f03f00b1730 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f00b0a70 con 0x7f03f00b4d60 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f00b0d50 con 0x7f03f014be90 2024-03-25T19:29:05.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:05.234 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.233+0000 7f03f86ba640 10 monclient(hunting): authenticate will time out at 6872.331058s 2024-03-25T19:29:05.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.247+0000 7f03f6c57640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f03f014be90 0x7f03f014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:48688/0 (socket says 172.21.15.60:48688) 2024-03-25T19:29:05.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.247+0000 7f03f6c57640 1 -- 172.21.15.60:0/1604337568 learned_addr learned my addr 172.21.15.60:0/1604337568 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:05.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.260+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330768081 0 0) 0x7f03f00b0d50 con 0x7f03f014be90 2024-03-25T19:29:05.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.260+0000 7f03f5454640 10 monclient(hunting): _negotiate 2024-03-25T19:29:05.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.260+0000 7f03f5454640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.260+0000 7f03f5454640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03dc003520 con 0x7f03f014be90 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486527165 0 0) 0x7f03f00af940 con 0x7f03f00b1730 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 10 monclient(hunting): _negotiate 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:29:05.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.261+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f00b0d50 con 0x7f03f00b1730 2024-03-25T19:29:05.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.281+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077136138 0 0) 0x7f03f00b0a70 con 0x7f03f00b4d60 2024-03-25T19:29:05.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.281+0000 7f03f5454640 10 monclient(hunting): _negotiate 2024-03-25T19:29:05.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.281+0000 7f03f5454640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:05.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.281+0000 7f03f5454640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:05.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.281+0000 7f03f5454640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:29:05.281 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.281+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f00af940 con 0x7f03f00b4d60 2024-03-25T19:29:05.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2609472332 0 0) 0x7f03f00af940 con 0x7f03f00b4d60 2024-03-25T19:29:05.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 >> v1:172.21.15.60:6790/0 conn(0x7f03f014be90 legacy=0x7f03f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:05.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 >> v1:172.21.15.60:6789/0 conn(0x7f03f00b1730 legacy=0x7f03f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 10 monclient: _finish_hunting 0 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 1 monclient: found mon.noname-c 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f00b1030 con 0x7f03f00b4d60 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 10 monclient: _finish_auth 0 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 10 monclient: _check_auth_tickets 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f03ec001570 con 0x7f03f00b4d60 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.291+0000 7f03f5454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.292+0000 7f03f5454640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.292+0000 7f03f5454640 10 monclient: dump: 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:05.292 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.292+0000 7f03f86ba640 5 monclient: authenticate success, global_id 4609 2024-03-25T19:29:05.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.292+0000 7f03f86ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:05.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03ec0016d0 con 0x7f03f00b4d60 2024-03-25T19:29:05.305 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f5454640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f5454640 1 -- 172.21.15.60:0/1604337568 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f03ec001ae0 con 0x7f03f00b4d60 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f5454640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f5454640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f5454640 10 monclient: dump: 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:05.306 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:05.307 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:05.307 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:05.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f86ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:05.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.305+0000 7f03f86ba640 4 set_mon_vals no callback set 2024-03-25T19:29:05.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.306+0000 7f03f86ba640 10 monclient: shutdown 2024-03-25T19:29:05.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.306+0000 7f03f86ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:05.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.306+0000 7f03f86ba640 1 -- 172.21.15.60:0/1604337568 >> v1:172.21.15.90:6789/0 conn(0x7f03f00b4d60 legacy=0x7f03f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:05.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.306+0000 7f03f86ba640 1 -- 172.21.15.60:0/1604337568 shutdown_connections 2024-03-25T19:29:05.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.306+0000 7f03f86ba640 1 -- 172.21.15.60:0/1604337568 >> 172.21.15.60:0/1604337568 conn(0x7f03f00617a0 msgr2=0x7f03f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:05.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.306+0000 7f03f86ba640 1 -- 172.21.15.60:0/1604337568 shutdown_connections 2024-03-25T19:29:05.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.307+0000 7f03f86ba640 1 -- 172.21.15.60:0/1604337568 wait complete. 2024-03-25T19:29:05.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 10 monclient: build_initial_monmap 2024-03-25T19:29:05.309 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 10 monclient: monmap: 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 1 librados: starting msgr at 2024-03-25T19:29:05.310 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 1 librados: starting objecter 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 1 Processor -- start 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 1 -- start start 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 1 librados: setting wanted keys 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 1 librados: calling monclient init 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.308+0000 7f03f86ba640 10 monclient: init 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 10 monclient(hunting): picked mon.noname-b con 0x7f03f00b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 10 monclient(hunting): picked mon.noname-a con 0x7f03f00b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 10 monclient(hunting): picked mon.noname-c con 0x7f03f014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f0141c80 con 0x7f03f00b4d60 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f01403f0 con 0x7f03f00b1730 2024-03-25T19:29:05.311 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f013dba0 con 0x7f03f014be90 2024-03-25T19:29:05.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:05.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.309+0000 7f03f86ba640 10 monclient(hunting): authenticate will time out at 6872.407363s 2024-03-25T19:29:05.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.315+0000 7f03f6456640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f03f00b1730 0x7f03f013c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:52382/0 (socket says 172.21.15.60:52382) 2024-03-25T19:29:05.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.315+0000 7f03f6456640 1 -- 172.21.15.60:0/2625579465 learned_addr learned my addr 172.21.15.60:0/2625579465 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:05.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.319+0000 7f03e6ffd640 10 client.?.objecter ms_handle_connect 0x7f03f00b1730 2024-03-25T19:29:05.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.319+0000 7f03e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:05.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.319+0000 7f03e6ffd640 10 client.?.objecter ms_handle_connect 0x7f03f00b4d60 2024-03-25T19:29:05.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.319+0000 7f03e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:05.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 10 client.?.objecter ms_handle_connect 0x7f03f014be90 2024-03-25T19:29:05.323 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:05.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250126772 0 0) 0x7f03f0141c80 con 0x7f03f00b4d60 2024-03-25T19:29:05.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:05.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:05.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:05.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:29:05.324 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.324+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d4004eb0 con 0x7f03f00b4d60 2024-03-25T19:29:05.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.328+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693610734 0 0) 0x7f03f01403f0 con 0x7f03f00b1730 2024-03-25T19:29:05.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.328+0000 7f03e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:05.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.328+0000 7f03e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:05.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.328+0000 7f03e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:05.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.328+0000 7f03e6ffd640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:29:05.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.328+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f0141c80 con 0x7f03f00b1730 2024-03-25T19:29:05.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.333+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1237649294 0 0) 0x7f03d4004eb0 con 0x7f03f00b4d60 2024-03-25T19:29:05.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.333+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03f01403f0 con 0x7f03f00b4d60 2024-03-25T19:29:05.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f03e0001990 con 0x7f03f00b4d60 2024-03-25T19:29:05.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:05.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:05.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:29:05.341 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:05.341 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109187727 0 0) 0x7f03f01403f0 con 0x7f03f00b4d60 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 >> v1:172.21.15.60:6790/0 conn(0x7f03f014be90 legacy=0x7f03f013e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 >> v1:172.21.15.90:6789/0 conn(0x7f03f00b1730 legacy=0x7f03f013c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 1 monclient: found mon.a 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:05.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f013e7d0 con 0x7f03f00b4d60 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 5 monclient: authenticate success, global_id 4623 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 10 monclient: _renew_subs 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f013ea20 con 0x7f03f00b4d60 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 10 client.4623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 10 monclient: _renew_subs 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f0141640 con 0x7f03f00b4d60 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.341+0000 7f03f86ba640 1 librados: init done 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.342+0000 7f03f86ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.342+0000 7f03f86ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:05.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.342+0000 7f03f86ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:05.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.342+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f00627b0 con 0x7f03f00b4d60 2024-03-25T19:29:05.661 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03e0003610 con 0x7f03f00b4d60 2024-03-25T19:29:05.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:05.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f03e00039e0 con 0x7f03f00b4d60 2024-03-25T19:29:05.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:05.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:05.662 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03e6ffd640 10 monclient: dump: 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:05.663 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:05.664 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:05.664 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:05.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03f4c53640 4 set_mon_vals no callback set 2024-03-25T19:29:05.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.662+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f03e0003c00 con 0x7f03f00b4d60 2024-03-25T19:29:05.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.663+0000 7f03f6456640 10 monclient: get_auth_request con 0x7f03d406dcb0 auth_method 0 2024-03-25T19:29:05.664 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.663+0000 7f03e6ffd640 10 client.4623.objecter ms_handle_connect 0x7f03d406dcb0 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f03e0053610 con 0x7f03f00b4d60 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 10 client.4623.objecter ms_dispatch 0x7f03f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 3 client.4623.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 3 client.4623.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 20 client.4623.objecter dump_active .. 0 homeless 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03e004f120 con 0x7f03f00b4d60 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:05.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.671+0000 7f03e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:05.891 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.891+0000 7f03f86ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:05.891 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.891+0000 7f03f86ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:05.891 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.891+0000 7f03f86ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:05.891 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.891+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03f013ebb0 con 0x7f03f00b4d60 2024-03-25T19:29:05.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.913+0000 7f03e6ffd640 1 -- 172.21.15.60:0/2625579465 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f03e0051050 con 0x7f03f00b4d60 2024-03-25T19:29:05.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.913+0000 7f03e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:05.913 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.913+0000 7f03e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:05.913 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:05.913 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:05.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 10 librados: watch_flush enter 2024-03-25T19:29:05.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 10 librados: watch_flush exit 2024-03-25T19:29:05.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 20 client.4623.objecter shutdown clearing up homeless session... 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 10 client.4623.objecter successfully canceled tick 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 >> v1:172.21.15.90:6800/1827638754 conn(0x7f03d406dcb0 legacy=0x7f03d4070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 10 monclient: shutdown 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 >> v1:172.21.15.60:6789/0 conn(0x7f03f00b4d60 legacy=0x7f03f013d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 shutdown_connections 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 >> 172.21.15.60:0/2625579465 conn(0x7f03f00617a0 msgr2=0x7f03f0143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 shutdown_connections 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 1 -- 172.21.15.60:0/2625579465 wait complete. 2024-03-25T19:29:05.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:05.916+0000 7f03f86ba640 1 librados: shutdown 2024-03-25T19:29:05.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:12.734 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:12.848 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.849+0000 7f4e38f72640 10 monclient: get_monmap_and_config 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.849+0000 7f4e38f72640 10 monclient: build_initial_monmap 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.849+0000 7f4e38f72640 10 monclient: monmap: 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:12.850037+0000 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:12.850037+0000 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:12.849 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:12.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 1 Processor -- start 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 1 -- start start 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient: init 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient(hunting): picked mon.noname-c con 0x7f4e340a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient(hunting): picked mon.noname-a con 0x7f4e3413f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient(hunting): picked mon.noname-b con 0x7f4e341430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e34142520 con 0x7f4e3413f710 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340b28a0 con 0x7f4e340a98f0 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340b2a30 con 0x7f4e341430c0 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:12.851 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.850+0000 7f4e38f72640 10 monclient(hunting): authenticate will time out at 6879.948735s 2024-03-25T19:29:12.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.861+0000 7f4e3259c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4e340a98f0 0x7f4e340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:60088/0 (socket says 172.21.15.60:60088) 2024-03-25T19:29:12.860 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.861+0000 7f4e3259c640 1 -- 172.21.15.60:0/3810412354 learned_addr learned my addr 172.21.15.60:0/3810412354 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:12.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.869+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937504075 0 0) 0x7f4e340b28a0 con 0x7f4e340a98f0 2024-03-25T19:29:12.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.869+0000 7f4e31d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:12.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.869+0000 7f4e31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:12.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.869+0000 7f4e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:12.869 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.869+0000 7f4e31d9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:29:12.869 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.869+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e20003520 con 0x7f4e340a98f0 2024-03-25T19:29:12.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.886+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127240194 0 0) 0x7f4e34142520 con 0x7f4e3413f710 2024-03-25T19:29:12.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.886+0000 7f4e31d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:29:12.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.886+0000 7f4e31d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:12.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.886+0000 7f4e31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:12.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.886+0000 7f4e31d9b640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:29:12.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.886+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e340b28a0 con 0x7f4e3413f710 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1347538086 0 0) 0x7f4e20003520 con 0x7f4e340a98f0 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 >> v1:172.21.15.60:6790/0 conn(0x7f4e341430c0 legacy=0x7f4e3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 >> v1:172.21.15.60:6789/0 conn(0x7f4e3413f710 legacy=0x7f4e341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 1 monclient: found mon.noname-c 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:29:12.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e340b44c0 con 0x7f4e340a98f0 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 10 monclient: _finish_auth 0 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4e24001690 con 0x7f4e340a98f0 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e31d9b640 10 monclient: dump: 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:12.889 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e38f72640 5 monclient: authenticate success, global_id 4615 2024-03-25T19:29:12.890 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.888+0000 7f4e38f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:12.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e240017f0 con 0x7f4e340a98f0 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e31d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e31d9b640 1 -- 172.21.15.60:0/3810412354 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4e24001c00 con 0x7f4e340a98f0 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e38f72640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e38f72640 4 set_mon_vals no callback set 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e31d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e31d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.897+0000 7f4e31d9b640 10 monclient: dump: 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:12.897 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:12.898 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:12.898 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:12.898 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:12.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.898+0000 7f4e38f72640 10 monclient: shutdown 2024-03-25T19:29:12.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.898+0000 7f4e38f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:12.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.898+0000 7f4e38f72640 1 -- 172.21.15.60:0/3810412354 >> v1:172.21.15.90:6789/0 conn(0x7f4e340a98f0 legacy=0x7f4e340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.898+0000 7f4e38f72640 1 -- 172.21.15.60:0/3810412354 shutdown_connections 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.898+0000 7f4e38f72640 1 -- 172.21.15.60:0/3810412354 >> 172.21.15.60:0/3810412354 conn(0x7f4e340b7ea0 msgr2=0x7f4e340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.898+0000 7f4e38f72640 1 -- 172.21.15.60:0/3810412354 shutdown_connections 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.898+0000 7f4e38f72640 1 -- 172.21.15.60:0/3810412354 wait complete. 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 10 monclient: build_initial_monmap 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 10 monclient: monmap: 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:12.900 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 1 librados: starting msgr at 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 1 librados: starting objecter 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 1 Processor -- start 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 1 -- start start 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 1 librados: setting wanted keys 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 1 librados: calling monclient init 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.899+0000 7f4e38f72640 10 monclient: init 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.900+0000 7f4e38f72640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:12.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.900+0000 7f4e38f72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:12.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.900+0000 7f4e38f72640 10 monclient(hunting): picked mon.noname-a con 0x7f4e340a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:12.902 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.900+0000 7f4e38f72640 10 monclient(hunting): picked mon.noname-b con 0x7f4e3413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.900+0000 7f4e38f72640 10 monclient(hunting): picked mon.noname-c con 0x7f4e341430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.900+0000 7f4e38f72640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e34138b40 con 0x7f4e340a98f0 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.900+0000 7f4e38f72640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e34138e20 con 0x7f4e3413f710 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.901+0000 7f4e38f72640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e34139100 con 0x7f4e341430c0 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.901+0000 7f4e38f72640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.901+0000 7f4e38f72640 10 monclient(hunting): authenticate will time out at 6879.998940s 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.904+0000 7f4e3259c640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f4e340a98f0 0x7f4e34137380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53438/0 (socket says 172.21.15.60:53438) 2024-03-25T19:29:12.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.904+0000 7f4e3259c640 1 -- 172.21.15.60:0/2471469682 learned_addr learned my addr 172.21.15.60:0/2471469682 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:12.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.906+0000 7f4e2affd640 10 client.?.objecter ms_handle_connect 0x7f4e340a98f0 2024-03-25T19:29:12.905 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.906+0000 7f4e2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:12.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.914+0000 7f4e2affd640 10 client.?.objecter ms_handle_connect 0x7f4e341430c0 2024-03-25T19:29:12.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.914+0000 7f4e2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:12.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.915+0000 7f4e2affd640 10 client.?.objecter ms_handle_connect 0x7f4e3413f710 2024-03-25T19:29:12.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.915+0000 7f4e2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:12.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566244930 0 0) 0x7f4e34138b40 con 0x7f4e340a98f0 2024-03-25T19:29:12.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:12.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:12.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:12.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:29:12.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e14004eb0 con 0x7f4e340a98f0 2024-03-25T19:29:12.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766935132 0 0) 0x7f4e34139100 con 0x7f4e341430c0 2024-03-25T19:29:12.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:12.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:12.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:12.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:29:12.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.924+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e34138b40 con 0x7f4e341430c0 2024-03-25T19:29:12.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.931+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201217155 0 0) 0x7f4e34138e20 con 0x7f4e3413f710 2024-03-25T19:29:12.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.931+0000 7f4e2affd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:12.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.931+0000 7f4e2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:12.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.931+0000 7f4e2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:12.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.931+0000 7f4e2affd640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:29:12.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.931+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e34139100 con 0x7f4e3413f710 2024-03-25T19:29:12.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.936+0000 7f4e3259c640 1 -- 172.21.15.60:0/2471469682 >> v1:172.21.15.60:6789/0 conn(0x7f4e340a98f0 legacy=0x7f4e34137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:29:12.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.936+0000 7f4e3259c640 1 -- 172.21.15.60:0/2471469682 >> v1:172.21.15.60:6789/0 conn(0x7f4e340a98f0 legacy=0x7f4e34137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:29:12.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.936+0000 7f4e3259c640 1 --1- 172.21.15.60:0/2471469682 >> v1:172.21.15.60:6789/0 conn(0x7f4e340a98f0 0x7f4e34137380 :-1 s=OPENED pgs=66287 cs=1 l=1).handle_message read tag failed 2024-03-25T19:29:12.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.936+0000 7f4e3259c640 1 --1- 172.21.15.60:0/2471469682 >> v1:172.21.15.60:6789/0 conn(0x7f4e340a98f0 0x7f4e34137380 :-1 s=OPENED pgs=66287 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:29:12.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.936+0000 7f4e2affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.60:6789/0 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.943+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3175374193 0 0) 0x7f4e34139100 con 0x7f4e3413f710 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.943+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e34138e20 con 0x7f4e3413f710 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.943+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4e1c001930 con 0x7f4e3413f710 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.943+0000 7f4e2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.943+0000 7f4e2affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.943+0000 7f4e2affd640 10 monclient(hunting): dump: 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:12.943 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:12.944 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:12.944 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:12.944 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:12.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.943+0000 7f4e2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.944+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87907775 0 0) 0x7f4e34138b40 con 0x7f4e341430c0 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.944+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e34139100 con 0x7f4e341430c0 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.944+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4e2c001b10 con 0x7f4e341430c0 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.944+0000 7f4e2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.944+0000 7f4e2affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.944+0000 7f4e2affd640 10 monclient(hunting): dump: 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:12.945 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:12.946 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:12.946 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:12.946 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 832718158 0 0) 0x7f4e34138e20 con 0x7f4e3413f710 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 >> v1:172.21.15.60:6790/0 conn(0x7f4e341430c0 legacy=0x7f4e341e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 >> v1:172.21.15.60:6789/0 conn(0x7f4e340a98f0 legacy=0x7f4e34137380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 1 monclient: found mon.b 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e341397a0 con 0x7f4e3413f710 2024-03-25T19:29:12.959 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 10 monclient: _finish_auth 0 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 10 monclient: _renew_subs 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e34136aa0 con 0x7f4e3413f710 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 10 monclient: _renew_subs 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3413c320 con 0x7f4e3413f710 2024-03-25T19:29:12.960 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.959+0000 7f4e38f72640 1 librados: init done 2024-03-25T19:29:12.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.960+0000 7f4e38f72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:12.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.960+0000 7f4e38f72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:12.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.960+0000 7f4e38f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:12.961 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:12.960+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e341395b0 con 0x7f4e3413f710 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.236+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e1c0035f0 con 0x7f4e3413f710 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.236+0000 7f4e2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.236+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4e1c003900 con 0x7f4e3413f710 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.236+0000 7f4e2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.236+0000 7f4e2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.236+0000 7f4e2affd640 10 monclient: dump: 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:13.236 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.236+0000 7f4e3159a640 4 set_mon_vals no callback set 2024-03-25T19:29:13.237 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.237+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f4e1c0198d0 con 0x7f4e3413f710 2024-03-25T19:29:13.238 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.238+0000 7f4e3259c640 10 monclient: get_auth_request con 0x7f4e1406df80 auth_method 0 2024-03-25T19:29:13.238 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.238+0000 7f4e2affd640 10 client.4618.objecter ms_handle_connect 0x7f4e1406df80 2024-03-25T19:29:13.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.242+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f4e1c053550 con 0x7f4e3413f710 2024-03-25T19:29:13.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.242+0000 7f4e2affd640 10 client.4618.objecter ms_dispatch 0x7f4e34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:13.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.242+0000 7f4e2affd640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:13.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.242+0000 7f4e2affd640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:13.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.242+0000 7f4e2affd640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T19:29:13.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.243+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e1c04f060 con 0x7f4e3413f710 2024-03-25T19:29:13.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.243+0000 7f4e2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:13.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.243+0000 7f4e2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:13.499 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.500+0000 7f4e38f72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:13.499 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.500+0000 7f4e38f72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:13.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.500+0000 7f4e38f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:13.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.500+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e34139930 con 0x7f4e3413f710 2024-03-25T19:29:13.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.534+0000 7f4e2affd640 1 -- 172.21.15.60:0/2471469682 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f4e1c050f90 con 0x7f4e3413f710 2024-03-25T19:29:13.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.535+0000 7f4e2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:13.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.535+0000 7f4e2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:13.535 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:13.535 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 10 librados: watch_flush enter 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 10 librados: watch_flush exit 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 10 client.4618.objecter successfully canceled tick 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 >> v1:172.21.15.90:6800/1827638754 conn(0x7f4e1406df80 legacy=0x7f4e14070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 10 monclient: shutdown 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 >> v1:172.21.15.90:6789/0 conn(0x7f4e3413f710 legacy=0x7f4e34137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 shutdown_connections 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 >> 172.21.15.60:0/2471469682 conn(0x7f4e340b7ea0 msgr2=0x7f4e3414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:13.537 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 shutdown_connections 2024-03-25T19:29:13.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 1 -- 172.21.15.60:0/2471469682 wait complete. 2024-03-25T19:29:13.538 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:13.537+0000 7f4e38f72640 1 librados: shutdown 2024-03-25T19:29:13.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:20.355 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.473+0000 7fa0dee7b640 10 monclient: get_monmap_and_config 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.473+0000 7fa0dee7b640 10 monclient: build_initial_monmap 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.473+0000 7fa0dee7b640 10 monclient: monmap: 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:20.474393+0000 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:20.474393+0000 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:20.473 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:20.474 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:20.474 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:20.474 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.474+0000 7fa0dee7b640 1 Processor -- start 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.474+0000 7fa0dee7b640 1 -- start start 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.474+0000 7fa0dee7b640 10 monclient: init 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa0d80a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa0d813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa0d81430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d8142520 con 0x7fa0d80a98f0 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d80b28a0 con 0x7fa0d81430c0 2024-03-25T19:29:20.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d80b2a30 con 0x7fa0d813f710 2024-03-25T19:29:20.476 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:20.476 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.475+0000 7fa0dee7b640 10 monclient(hunting): authenticate will time out at 6887.573148s 2024-03-25T19:29:20.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.489+0000 7fa0dcc17640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fa0d80a98f0 0x7fa0d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:32852/0 (socket says 172.21.15.60:32852) 2024-03-25T19:29:20.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.489+0000 7fa0dcc17640 1 -- 172.21.15.60:0/1799217216 learned_addr learned my addr 172.21.15.60:0/1799217216 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:20.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.497+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052580556 0 0) 0x7fa0d8142520 con 0x7fa0d80a98f0 2024-03-25T19:29:20.496 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.497+0000 7fa0d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:20.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.497+0000 7fa0d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:20.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.497+0000 7fa0d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:20.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.497+0000 7fa0d77fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:29:20.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.497+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c8003520 con 0x7fa0d80a98f0 2024-03-25T19:29:20.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.506+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135427862 0 0) 0x7fa0d80b28a0 con 0x7fa0d81430c0 2024-03-25T19:29:20.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.507+0000 7fa0d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:20.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.507+0000 7fa0d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:20.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.507+0000 7fa0d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:20.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.507+0000 7fa0d77fe640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:29:20.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.507+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d8142520 con 0x7fa0d81430c0 2024-03-25T19:29:20.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.510+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097668529 0 0) 0x7fa0d80b2a30 con 0x7fa0d813f710 2024-03-25T19:29:20.510 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.510+0000 7fa0d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:29:20.510 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.510+0000 7fa0d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:20.510 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.510+0000 7fa0d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:20.510 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.510+0000 7fa0d77fe640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:29:20.510 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.510+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d80b28a0 con 0x7fa0d813f710 2024-03-25T19:29:20.511 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 206338033 0 0) 0x7fa0c8003520 con 0x7fa0d80a98f0 2024-03-25T19:29:20.511 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 >> v1:172.21.15.60:6790/0 conn(0x7fa0d813f710 legacy=0x7fa0d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:20.511 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 >> v1:172.21.15.90:6789/0 conn(0x7fa0d81430c0 legacy=0x7fa0d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:20.511 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:29:20.511 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 1 monclient: found mon.noname-a 2024-03-25T19:29:20.511 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d80b44c0 con 0x7fa0d80a98f0 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa0cc001690 con 0x7fa0d80a98f0 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0d77fe640 10 monclient: dump: 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:20.512 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:20.513 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:20.513 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:20.513 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:20.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0dee7b640 5 monclient: authenticate success, global_id 4632 2024-03-25T19:29:20.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.511+0000 7fa0dee7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:20.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0cc0017f0 con 0x7fa0d80a98f0 2024-03-25T19:29:20.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:20.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0d77fe640 1 -- 172.21.15.60:0/1799217216 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa0cc001c00 con 0x7fa0d80a98f0 2024-03-25T19:29:20.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:20.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:20.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0d77fe640 10 monclient: dump: 2024-03-25T19:29:20.543 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0dee7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:20.544 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.543+0000 7fa0dee7b640 4 set_mon_vals no callback set 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.544+0000 7fa0dee7b640 10 monclient: shutdown 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.544+0000 7fa0dee7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.544+0000 7fa0dee7b640 1 -- 172.21.15.60:0/1799217216 >> v1:172.21.15.60:6789/0 conn(0x7fa0d80a98f0 legacy=0x7fa0d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.544+0000 7fa0dee7b640 1 -- 172.21.15.60:0/1799217216 shutdown_connections 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.544+0000 7fa0dee7b640 1 -- 172.21.15.60:0/1799217216 >> 172.21.15.60:0/1799217216 conn(0x7fa0d80b7ea0 msgr2=0x7fa0d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.544+0000 7fa0dee7b640 1 -- 172.21.15.60:0/1799217216 shutdown_connections 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.544+0000 7fa0dee7b640 1 -- 172.21.15.60:0/1799217216 wait complete. 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.545+0000 7fa0dee7b640 10 monclient: build_initial_monmap 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.545+0000 7fa0dee7b640 10 monclient: monmap: 2024-03-25T19:29:20.547 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.545+0000 7fa0dee7b640 1 librados: starting msgr at 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.545+0000 7fa0dee7b640 1 librados: starting objecter 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.546+0000 7fa0dee7b640 1 Processor -- start 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.546+0000 7fa0dee7b640 1 -- start start 2024-03-25T19:29:20.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.546+0000 7fa0dee7b640 1 librados: setting wanted keys 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.546+0000 7fa0dee7b640 1 librados: calling monclient init 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.546+0000 7fa0dee7b640 10 monclient: init 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 10 monclient(hunting): picked mon.noname-c con 0x7fa0d80a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 10 monclient(hunting): picked mon.noname-a con 0x7fa0d813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 10 monclient(hunting): picked mon.noname-b con 0x7fa0d81430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d8138970 con 0x7fa0d813f710 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d81d7810 con 0x7fa0d81430c0 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0d8139990 con 0x7fa0d80a98f0 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:20.549 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.547+0000 7fa0dee7b640 10 monclient(hunting): authenticate will time out at 6887.645251s 2024-03-25T19:29:20.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.557+0000 7fa0dd418640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fa0d81430c0 0x7fa0d81d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:51100/0 (socket says 172.21.15.60:51100) 2024-03-25T19:29:20.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.557+0000 7fa0dd418640 1 -- 172.21.15.60:0/3470454232 learned_addr learned my addr 172.21.15.60:0/3470454232 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:20.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.559+0000 7fa0d57fa640 10 client.?.objecter ms_handle_connect 0x7fa0d81430c0 2024-03-25T19:29:20.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.559+0000 7fa0d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:20.562 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.563+0000 7fa0d57fa640 10 client.?.objecter ms_handle_connect 0x7fa0d80a98f0 2024-03-25T19:29:20.563 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.563+0000 7fa0d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:20.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.619+0000 7fa0d57fa640 10 client.?.objecter ms_handle_connect 0x7fa0d813f710 2024-03-25T19:29:20.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.619+0000 7fa0d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:20.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.628+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271521478 0 0) 0x7fa0d8138970 con 0x7fa0d813f710 2024-03-25T19:29:20.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.628+0000 7fa0d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:20.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.628+0000 7fa0d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:20.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.628+0000 7fa0d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:20.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.628+0000 7fa0d57fa640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:29:20.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.628+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0bc004eb0 con 0x7fa0d813f710 2024-03-25T19:29:20.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.659+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1389323729 0 0) 0x7fa0bc004eb0 con 0x7fa0d813f710 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.659+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0d8138970 con 0x7fa0d813f710 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.659+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa0c40018f0 con 0x7fa0d813f710 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.659+0000 7fa0d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.659+0000 7fa0d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.659+0000 7fa0d57fa640 10 monclient(hunting): dump: 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:20.659 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:20.660 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:20.660 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.659+0000 7fa0d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:20.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.679+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070091242 0 0) 0x7fa0d81d7810 con 0x7fa0d81430c0 2024-03-25T19:29:20.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.679+0000 7fa0d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:29:20.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.679+0000 7fa0d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:20.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.679+0000 7fa0d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:20.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.679+0000 7fa0d57fa640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:29:20.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.679+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0bc004eb0 con 0x7fa0d81430c0 2024-03-25T19:29:20.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2888333044 0 0) 0x7fa0d8138970 con 0x7fa0d813f710 2024-03-25T19:29:20.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 >> v1:172.21.15.60:6790/0 conn(0x7fa0d80a98f0 legacy=0x7fa0d8138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:20.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 >> v1:172.21.15.90:6789/0 conn(0x7fa0d81430c0 legacy=0x7fa0d81d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 1 monclient: found mon.a 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d81d8440 con 0x7fa0d813f710 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 10 monclient: _finish_auth 0 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0dee7b640 5 monclient: authenticate success, global_id 4635 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0dee7b640 10 monclient: _renew_subs 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0dee7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.683+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d81d41e0 con 0x7fa0d813f710 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.684+0000 7fa0dee7b640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.684+0000 7fa0dee7b640 10 monclient: _renew_subs 2024-03-25T19:29:20.684 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.684+0000 7fa0dee7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:20.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.684+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d8138d80 con 0x7fa0d813f710 2024-03-25T19:29:20.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.684+0000 7fa0dee7b640 1 librados: init done 2024-03-25T19:29:20.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.685+0000 7fa0dee7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:20.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.685+0000 7fa0dee7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:20.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.685+0000 7fa0dee7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:20.686 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:20.685+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d80627b0 con 0x7fa0d813f710 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.024+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0c4003610 con 0x7fa0d813f710 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.024+0000 7fa0d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.024+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fa0c4003a60 con 0x7fa0d813f710 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.024+0000 7fa0d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.024+0000 7fa0d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.024+0000 7fa0d57fa640 10 monclient: dump: 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:21.024 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:21.025 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:21.025 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:21.025 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:21.025 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:21.025 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:21.025 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:21.025 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.024+0000 7fa0d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:29:21.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.025+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fa0c4003c80 con 0x7fa0d813f710 2024-03-25T19:29:21.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.026+0000 7fa0dcc17640 10 monclient: get_auth_request con 0x7fa0bc06dd90 auth_method 0 2024-03-25T19:29:21.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.026+0000 7fa0d57fa640 10 client.4635.objecter ms_handle_connect 0x7fa0bc06dd90 2024-03-25T19:29:21.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fa0c40535f0 con 0x7fa0d813f710 2024-03-25T19:29:21.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 10 client.4635.objecter ms_dispatch 0x7fa0d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:21.032 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:21.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:21.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T19:29:21.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0c404f100 con 0x7fa0d813f710 2024-03-25T19:29:21.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:21.033 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.033+0000 7fa0d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:21.152 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:21.863 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.157 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:21.869 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.193 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:21.904 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.211 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:21.865 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.222 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:21.933 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.225 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:21.879 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.229 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:21.941 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.248+0000 7fa0dee7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:21.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.248+0000 7fa0dee7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:21.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.248+0000 7fa0dee7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:21.248 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.248+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0d81d4370 con 0x7fa0d813f710 2024-03-25T19:29:21.257 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:21.911 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.261 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:21.972 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:21.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.315+0000 7fa0d57fa640 1 -- 172.21.15.60:0/3470454232 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fa0c4051030 con 0x7fa0d813f710 2024-03-25T19:29:21.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.315+0000 7fa0d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:21.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.315+0000 7fa0d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:21.315 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:21.315 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:21.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 10 librados: watch_flush enter 2024-03-25T19:29:21.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 10 librados: watch_flush exit 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 10 client.4635.objecter successfully canceled tick 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 >> v1:172.21.15.90:6800/1827638754 conn(0x7fa0bc06dd90 legacy=0x7fa0bc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 10 monclient: shutdown 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.317+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 >> v1:172.21.15.60:6789/0 conn(0x7fa0d813f710 legacy=0x7fa0d81d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.318+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 shutdown_connections 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.318+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 >> 172.21.15.60:0/3470454232 conn(0x7fa0d80b7ea0 msgr2=0x7fa0d813c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.318+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 shutdown_connections 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.318+0000 7fa0dee7b640 1 -- 172.21.15.60:0/3470454232 wait complete. 2024-03-25T19:29:21.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:21.318+0000 7fa0dee7b640 1 librados: shutdown 2024-03-25T19:29:21.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:28.135 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:28.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.252+0000 7f06c616c640 10 monclient: get_monmap_and_config 2024-03-25T19:29:28.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.252+0000 7f06c616c640 10 monclient: build_initial_monmap 2024-03-25T19:29:28.251 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.252+0000 7f06c616c640 10 monclient: monmap: 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:28.252766+0000 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:28.252766+0000 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:28.252 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:28.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.252+0000 7f06c616c640 1 Processor -- start 2024-03-25T19:29:28.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.252+0000 7f06c616c640 1 -- start start 2024-03-25T19:29:28.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.252+0000 7f06c616c640 10 monclient: init 2024-03-25T19:29:28.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:28.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:28.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 10 monclient(hunting): picked mon.noname-c con 0x7f06c00b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:28.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 10 monclient(hunting): picked mon.noname-b con 0x7f06c00b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 10 monclient(hunting): picked mon.noname-a con 0x7f06c014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c00af940 con 0x7f06c014be90 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c00b0a70 con 0x7f06c00b4d60 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c00b0d50 con 0x7f06c00b1730 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.253+0000 7f06c616c640 10 monclient(hunting): authenticate will time out at 6895.351531s 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.254+0000 7f06bf7fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f06c00b4d60 0x7f06c013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:48070/0 (socket says 172.21.15.60:48070) 2024-03-25T19:29:28.254 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.255+0000 7f06bf7fe640 1 -- 172.21.15.60:0/2857903582 learned_addr learned my addr 172.21.15.60:0/2857903582 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:28.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.265+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898372068 0 0) 0x7f06c00b0a70 con 0x7f06c00b4d60 2024-03-25T19:29:28.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.265+0000 7f06beffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:28.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.265+0000 7f06beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:28.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.265+0000 7f06beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:28.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.265+0000 7f06beffd640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:29:28.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.265+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06ac003520 con 0x7f06c00b4d60 2024-03-25T19:29:28.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.280+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880079794 0 0) 0x7f06c00af940 con 0x7f06c014be90 2024-03-25T19:29:28.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.280+0000 7f06beffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:28.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.280+0000 7f06beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:28.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.280+0000 7f06beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:28.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.280+0000 7f06beffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:29:28.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.280+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c00b0a70 con 0x7f06c014be90 2024-03-25T19:29:28.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3148421394 0 0) 0x7f06ac003520 con 0x7f06c00b4d60 2024-03-25T19:29:28.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 >> v1:172.21.15.60:6790/0 conn(0x7f06c00b1730 legacy=0x7f06c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 >> v1:172.21.15.60:6789/0 conn(0x7f06c014be90 legacy=0x7f06c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 1 monclient: found mon.noname-c 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c00b1030 con 0x7f06c00b4d60 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f06b0001690 con 0x7f06c00b4d60 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:28.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06beffd640 10 monclient: dump: 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06c616c640 5 monclient: authenticate success, global_id 4627 2024-03-25T19:29:28.291 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.290+0000 7f06c616c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06b00017f0 con 0x7f06c00b4d60 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06beffd640 1 -- 172.21.15.60:0/2857903582 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f06b0001c00 con 0x7f06c00b4d60 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06c616c640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06c616c640 4 set_mon_vals no callback set 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:28.297 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.297+0000 7f06beffd640 10 monclient: dump: 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:28.298 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.298+0000 7f06c616c640 10 monclient: shutdown 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.298+0000 7f06c616c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.298+0000 7f06c616c640 1 -- 172.21.15.60:0/2857903582 >> v1:172.21.15.90:6789/0 conn(0x7f06c00b4d60 legacy=0x7f06c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.298+0000 7f06c616c640 1 -- 172.21.15.60:0/2857903582 shutdown_connections 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.298+0000 7f06c616c640 1 -- 172.21.15.60:0/2857903582 >> 172.21.15.60:0/2857903582 conn(0x7f06c00617a0 msgr2=0x7f06c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.298+0000 7f06c616c640 1 -- 172.21.15.60:0/2857903582 shutdown_connections 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.298+0000 7f06c616c640 1 -- 172.21.15.60:0/2857903582 wait complete. 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.299+0000 7f06c616c640 10 monclient: build_initial_monmap 2024-03-25T19:29:28.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.299+0000 7f06c616c640 10 monclient: monmap: 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.299+0000 7f06c616c640 1 librados: starting msgr at 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.299+0000 7f06c616c640 1 librados: starting objecter 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.299+0000 7f06c616c640 1 Processor -- start 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 1 -- start start 2024-03-25T19:29:28.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 1 librados: setting wanted keys 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 1 librados: calling monclient init 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient: init 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient(hunting): picked mon.noname-a con 0x7f06c00b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient(hunting): picked mon.noname-b con 0x7f06c00b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient(hunting): picked mon.noname-c con 0x7f06c014be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c01458e0 con 0x7f06c00b1730 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c01dac00 con 0x7f06c00b4d60 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c0136310 con 0x7f06c014be90 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:28.302 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.300+0000 7f06c616c640 10 monclient(hunting): authenticate will time out at 6895.398694s 2024-03-25T19:29:28.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.315+0000 7f06b6ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f06c00b4d60 0x7f06c01370b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:48096/0 (socket says 172.21.15.60:48096) 2024-03-25T19:29:28.315 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.315+0000 7f06b6ffd640 1 -- 172.21.15.60:0/3366805580 learned_addr learned my addr 172.21.15.60:0/3366805580 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:28.317 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.318+0000 7f06bcff9640 10 client.?.objecter ms_handle_connect 0x7f06c00b4d60 2024-03-25T19:29:28.318 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.318+0000 7f06bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:28.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.326+0000 7f06bcff9640 10 client.?.objecter ms_handle_connect 0x7f06c014be90 2024-03-25T19:29:28.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.326+0000 7f06bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:28.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.329+0000 7f06bcff9640 10 client.?.objecter ms_handle_connect 0x7f06c00b1730 2024-03-25T19:29:28.329 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.329+0000 7f06bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:28.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.330+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355030964 0 0) 0x7f06c01dac00 con 0x7f06c00b4d60 2024-03-25T19:29:28.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.330+0000 7f06bcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:28.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.330+0000 7f06bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:28.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.330+0000 7f06bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:28.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.330+0000 7f06bcff9640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:29:28.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.330+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a0004eb0 con 0x7f06c00b4d60 2024-03-25T19:29:28.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.343+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175156221 0 0) 0x7f06c0136310 con 0x7f06c014be90 2024-03-25T19:29:28.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.343+0000 7f06bcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:28.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.343+0000 7f06bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:28.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.343+0000 7f06bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:28.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.343+0000 7f06bcff9640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:29:28.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.343+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c01dac00 con 0x7f06c014be90 2024-03-25T19:29:28.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.345+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564138980 0 0) 0x7f06c01458e0 con 0x7f06c00b1730 2024-03-25T19:29:28.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.345+0000 7f06bcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:29:28.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.345+0000 7f06bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:28.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.345+0000 7f06bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:28.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.345+0000 7f06bcff9640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:29:28.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.345+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c0136310 con 0x7f06c00b1730 2024-03-25T19:29:28.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.346+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1621316365 0 0) 0x7f06a0004eb0 con 0x7f06c00b4d60 2024-03-25T19:29:28.345 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.346+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06c01458e0 con 0x7f06c00b4d60 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.346+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f06a8004080 con 0x7f06c00b4d60 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.346+0000 7f06bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.346+0000 7f06bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.346+0000 7f06bcff9640 10 monclient(hunting): dump: 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:28.346 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:28.347 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.346+0000 7f06bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:28.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2858441285 0 0) 0x7f06c01458e0 con 0x7f06c00b4d60 2024-03-25T19:29:28.364 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 >> v1:172.21.15.60:6790/0 conn(0x7f06c014be90 legacy=0x7f06c01db120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 >> v1:172.21.15.60:6789/0 conn(0x7f06c00b1730 legacy=0x7f06c0142aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 1 monclient: found mon.b 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c0136af0 con 0x7f06c00b4d60 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 10 monclient: _finish_auth 0 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06c616c640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06c616c640 10 monclient: _renew_subs 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06c616c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.364+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c0143900 con 0x7f06c00b4d60 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.365+0000 7f06c616c640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:28.365 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.365+0000 7f06c616c640 10 monclient: _renew_subs 2024-03-25T19:29:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.365+0000 7f06c616c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.365+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c0143390 con 0x7f06c00b4d60 2024-03-25T19:29:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.365+0000 7f06c616c640 1 librados: init done 2024-03-25T19:29:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.366+0000 7f06c616c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.366+0000 7f06c616c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.366+0000 7f06c616c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:28.366 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.366+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c01401f0 con 0x7f06c00b4d60 2024-03-25T19:29:28.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.634+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06a8000ed0 con 0x7f06c00b4d60 2024-03-25T19:29:28.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.634+0000 7f06bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:28.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.634+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f06a80012a0 con 0x7f06c00b4d60 2024-03-25T19:29:28.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.634+0000 7f06bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:28.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.634+0000 7f06bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:28.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.634+0000 7f06bcff9640 10 monclient: dump: 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:28.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.634+0000 7f06be7fc640 4 set_mon_vals no callback set 2024-03-25T19:29:28.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.635+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f06a80014c0 con 0x7f06c00b4d60 2024-03-25T19:29:28.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.636+0000 7f06bf7fe640 10 monclient: get_auth_request con 0x7f06a006ddf0 auth_method 0 2024-03-25T19:29:28.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.636+0000 7f06bcff9640 10 client.4630.objecter ms_handle_connect 0x7f06a006ddf0 2024-03-25T19:29:28.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.641+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f06a8054750 con 0x7f06c00b4d60 2024-03-25T19:29:28.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.641+0000 7f06bcff9640 10 client.4630.objecter ms_dispatch 0x7f06c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:28.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.641+0000 7f06bcff9640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:28.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.641+0000 7f06bcff9640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:28.641 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.641+0000 7f06bcff9640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T19:29:28.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.642+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06a8050260 con 0x7f06c00b4d60 2024-03-25T19:29:28.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.642+0000 7f06bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:28.642 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.642+0000 7f06bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:28.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.864+0000 7f06c616c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:28.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.864+0000 7f06c616c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:28.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.864+0000 7f06c616c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:28.864 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.864+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06c00627b0 con 0x7f06c00b4d60 2024-03-25T19:29:28.897 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.898+0000 7f06bcff9640 1 -- 172.21.15.60:0/3366805580 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f06a8052190 con 0x7f06c00b4d60 2024-03-25T19:29:28.898 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.898+0000 7f06bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:28.898 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.898+0000 7f06bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:28.898 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:28.898 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:28.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 10 librados: watch_flush enter 2024-03-25T19:29:28.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 10 librados: watch_flush exit 2024-03-25T19:29:28.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 10 client.4630.objecter successfully canceled tick 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 >> v1:172.21.15.90:6800/1827638754 conn(0x7f06a006ddf0 legacy=0x7f06a00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 10 monclient: shutdown 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 >> v1:172.21.15.90:6789/0 conn(0x7f06c00b4d60 legacy=0x7f06c01370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 shutdown_connections 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.900+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 >> 172.21.15.60:0/3366805580 conn(0x7f06c00617a0 msgr2=0x7f06c013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.901+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 shutdown_connections 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.901+0000 7f06c616c640 1 -- 172.21.15.60:0/3366805580 wait complete. 2024-03-25T19:29:28.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:28.901+0000 7f06c616c640 1 librados: shutdown 2024-03-25T19:29:28.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:35.719 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:35.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.837+0000 7f0bc0396640 10 monclient: get_monmap_and_config 2024-03-25T19:29:35.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.837+0000 7f0bc0396640 10 monclient: build_initial_monmap 2024-03-25T19:29:35.837 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.837+0000 7f0bc0396640 10 monclient: monmap: 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:35.838769+0000 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:35.838769+0000 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:35.838 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.837+0000 7f0bc0396640 1 Processor -- start 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.837+0000 7f0bc0396640 1 -- start start 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.837+0000 7f0bc0396640 10 monclient: init 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 10 monclient(hunting): picked mon.noname-c con 0x7f0bb80a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 10 monclient(hunting): picked mon.noname-b con 0x7f0bb813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 10 monclient(hunting): picked mon.noname-a con 0x7f0bb81430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8142520 con 0x7f0bb81430c0 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb80b28a0 con 0x7f0bb80a98f0 2024-03-25T19:29:35.842 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb80b2a30 con 0x7f0bb813f710 2024-03-25T19:29:35.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:35.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.838+0000 7f0bc0396640 10 monclient(hunting): authenticate will time out at 6902.937716s 2024-03-25T19:29:35.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.841+0000 7f0bbe132640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0bb80a98f0 0x7f0bb80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:46214/0 (socket says 172.21.15.60:46214) 2024-03-25T19:29:35.843 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.841+0000 7f0bbe132640 1 -- 172.21.15.60:0/3563128268 learned_addr learned my addr 172.21.15.60:0/3563128268 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:35.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.854+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686286196 0 0) 0x7f0bb80b28a0 con 0x7f0bb80a98f0 2024-03-25T19:29:35.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.854+0000 7f0bbd130640 10 monclient(hunting): _negotiate 2024-03-25T19:29:35.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.854+0000 7f0bbd130640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:35.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.854+0000 7f0bbd130640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:35.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.854+0000 7f0bbd130640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:29:35.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.854+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba4003520 con 0x7f0bb80a98f0 2024-03-25T19:29:35.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.870+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 400753744 0 0) 0x7f0ba4003520 con 0x7f0bb80a98f0 2024-03-25T19:29:35.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 >> v1:172.21.15.60:6790/0 conn(0x7f0bb813f710 legacy=0x7f0bb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:35.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 >> v1:172.21.15.60:6789/0 conn(0x7f0bb81430c0 legacy=0x7f0bb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:35.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 10 monclient: _finish_hunting 0 2024-03-25T19:29:35.871 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 1 monclient: found mon.noname-c 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80b44c0 con 0x7f0bb80a98f0 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 10 monclient: _finish_auth 0 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 10 monclient: _check_auth_tickets 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f0bb4001690 con 0x7f0bb80a98f0 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bbd130640 10 monclient: dump: 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:35.872 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bc0396640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:29:35.873 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.871+0000 7f0bc0396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:35.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bb40017f0 con 0x7f0bb80a98f0 2024-03-25T19:29:35.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bbd130640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:35.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bbd130640 1 -- 172.21.15.60:0/3563128268 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f0bb4001c00 con 0x7f0bb80a98f0 2024-03-25T19:29:35.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bbd130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:35.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bbd130640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:35.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bbd130640 10 monclient: dump: 2024-03-25T19:29:35.880 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bc0396640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:35.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.879+0000 7f0bc0396640 4 set_mon_vals no callback set 2024-03-25T19:29:35.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.880+0000 7f0bc0396640 10 monclient: shutdown 2024-03-25T19:29:35.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.880+0000 7f0bc0396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:35.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.880+0000 7f0bc0396640 1 -- 172.21.15.60:0/3563128268 >> v1:172.21.15.90:6789/0 conn(0x7f0bb80a98f0 legacy=0x7f0bb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:35.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.881+0000 7f0bc0396640 1 -- 172.21.15.60:0/3563128268 shutdown_connections 2024-03-25T19:29:35.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.881+0000 7f0bc0396640 1 -- 172.21.15.60:0/3563128268 >> 172.21.15.60:0/3563128268 conn(0x7f0bb80b7ea0 msgr2=0x7f0bb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:35.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.881+0000 7f0bc0396640 1 -- 172.21.15.60:0/3563128268 shutdown_connections 2024-03-25T19:29:35.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.881+0000 7f0bc0396640 1 -- 172.21.15.60:0/3563128268 wait complete. 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.882+0000 7f0bc0396640 10 monclient: build_initial_monmap 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.882+0000 7f0bc0396640 10 monclient: monmap: 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:35.884 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 librados: starting msgr at 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 librados: starting objecter 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 Processor -- start 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 -- start start 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 librados: setting wanted keys 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 librados: calling monclient init 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient: init 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient(hunting): picked mon.noname-a con 0x7f0bb80a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient(hunting): picked mon.noname-b con 0x7f0bb813f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient(hunting): picked mon.noname-c con 0x7f0bb81430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8138b20 con 0x7f0bb80a98f0 2024-03-25T19:29:35.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8137a80 con 0x7f0bb813f710 2024-03-25T19:29:35.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb8139880 con 0x7f0bb81430c0 2024-03-25T19:29:35.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:35.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.883+0000 7f0bc0396640 10 monclient(hunting): authenticate will time out at 6902.982767s 2024-03-25T19:29:35.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.894+0000 7f0bbd931640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f0bb813f710 0x7f0bb8139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:46248/0 (socket says 172.21.15.60:46248) 2024-03-25T19:29:35.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.894+0000 7f0bbd931640 1 -- 172.21.15.60:0/339724731 learned_addr learned my addr 172.21.15.60:0/339724731 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:35.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.895+0000 7f0baeffd640 10 client.?.objecter ms_handle_connect 0x7f0bb813f710 2024-03-25T19:29:35.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.895+0000 7f0baeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:35.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.901+0000 7f0baeffd640 10 client.?.objecter ms_handle_connect 0x7f0bb81430c0 2024-03-25T19:29:35.901 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.901+0000 7f0baeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:35.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.904+0000 7f0baeffd640 10 client.?.objecter ms_handle_connect 0x7f0bb80a98f0 2024-03-25T19:29:35.904 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.904+0000 7f0baeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:35.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.907+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451917589 0 0) 0x7f0bb8137a80 con 0x7f0bb813f710 2024-03-25T19:29:35.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.907+0000 7f0baeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:35.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.907+0000 7f0baeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:35.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.907+0000 7f0baeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:35.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.907+0000 7f0baeffd640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:29:35.908 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.908+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b9c004eb0 con 0x7f0bb813f710 2024-03-25T19:29:35.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.914+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998426076 0 0) 0x7f0bb8138b20 con 0x7f0bb80a98f0 2024-03-25T19:29:35.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.914+0000 7f0baeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:35.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.914+0000 7f0baeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:35.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.914+0000 7f0baeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:35.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.914+0000 7f0baeffd640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:29:35.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.914+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb8137a80 con 0x7f0bb80a98f0 2024-03-25T19:29:35.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.918+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463119444 0 0) 0x7f0bb8139880 con 0x7f0bb81430c0 2024-03-25T19:29:35.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.918+0000 7f0baeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:29:35.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.918+0000 7f0baeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:35.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.918+0000 7f0baeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:35.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.918+0000 7f0baeffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:29:35.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.918+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb8138b20 con 0x7f0bb81430c0 2024-03-25T19:29:35.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.924+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 966007388 0 0) 0x7f0b9c004eb0 con 0x7f0bb813f710 2024-03-25T19:29:35.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.924+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bb8139880 con 0x7f0bb813f710 2024-03-25T19:29:35.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.924+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f0ba8001a30 con 0x7f0bb813f710 2024-03-25T19:29:35.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.924+0000 7f0baeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:35.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.924+0000 7f0baeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:29:35.924 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.924+0000 7f0baeffd640 10 monclient(hunting): dump: 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:35.925 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.924+0000 7f0baeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:35.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4055510996 0 0) 0x7f0bb8139880 con 0x7f0bb813f710 2024-03-25T19:29:35.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 >> v1:172.21.15.60:6790/0 conn(0x7f0bb81430c0 legacy=0x7f0bb81e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 >> v1:172.21.15.60:6789/0 conn(0x7f0bb80a98f0 legacy=0x7f0bb8137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 1 monclient: found mon.b 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8139a10 con 0x7f0bb813f710 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 10 monclient: _finish_auth 0 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0baeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 10 monclient: _renew_subs 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb81dab00 con 0x7f0bb813f710 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:35.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 10 monclient: _renew_subs 2024-03-25T19:29:35.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:35.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb813b4f0 con 0x7f0bb813f710 2024-03-25T19:29:35.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.943+0000 7f0bc0396640 1 librados: init done 2024-03-25T19:29:35.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.945+0000 7f0bc0396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:35.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.945+0000 7f0bc0396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:35.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.945+0000 7f0bc0396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:35.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:35.945+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb81dae00 con 0x7f0bb813f710 2024-03-25T19:29:36.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.225+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba8001c70 con 0x7f0bb813f710 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.225+0000 7f0baeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.225+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f0ba8003a60 con 0x7f0bb813f710 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.225+0000 7f0baeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.226+0000 7f0baeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.226+0000 7f0baeffd640 10 monclient: dump: 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:36.227 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:36.228 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:36.228 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:36.228 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:36.228 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:36.228 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:36.228 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:36.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.226+0000 7f0bbc92f640 4 set_mon_vals no callback set 2024-03-25T19:29:36.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.226+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f0ba8003c40 con 0x7f0bb813f710 2024-03-25T19:29:36.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.227+0000 7f0bbe132640 10 monclient: get_auth_request con 0x7f0b9c06ddf0 auth_method 0 2024-03-25T19:29:36.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.227+0000 7f0baeffd640 10 client.4636.objecter ms_handle_connect 0x7f0b9c06ddf0 2024-03-25T19:29:36.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.231+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f0ba80548b0 con 0x7f0bb813f710 2024-03-25T19:29:36.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.231+0000 7f0baeffd640 10 client.4636.objecter ms_dispatch 0x7f0bb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:36.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.231+0000 7f0baeffd640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:36.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.231+0000 7f0baeffd640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:36.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.231+0000 7f0baeffd640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T19:29:36.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.232+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ba80503c0 con 0x7f0bb813f710 2024-03-25T19:29:36.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.232+0000 7f0baeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:36.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.232+0000 7f0baeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:36.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.488+0000 7f0bc0396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:36.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.488+0000 7f0bc0396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:36.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.488+0000 7f0bc0396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:29:36.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.488+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bb80b3e80 con 0x7f0bb813f710 2024-03-25T19:29:36.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.520+0000 7f0baeffd640 1 -- 172.21.15.60:0/339724731 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f0ba80522f0 con 0x7f0bb813f710 2024-03-25T19:29:36.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.520+0000 7f0baeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:36.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.520+0000 7f0baeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:36.521 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:36.521 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.522+0000 7f0bc0396640 10 librados: watch_flush enter 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.522+0000 7f0bc0396640 10 librados: watch_flush exit 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.522+0000 7f0bc0396640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.522+0000 7f0bc0396640 10 client.4636.objecter successfully canceled tick 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 >> v1:172.21.15.90:6800/1827638754 conn(0x7f0b9c06ddf0 legacy=0x7f0b9c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 10 monclient: shutdown 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 >> v1:172.21.15.90:6789/0 conn(0x7f0bb813f710 legacy=0x7f0bb8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 shutdown_connections 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 >> 172.21.15.60:0/339724731 conn(0x7f0bb80b7ea0 msgr2=0x7f0bb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 shutdown_connections 2024-03-25T19:29:36.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 1 -- 172.21.15.60:0/339724731 wait complete. 2024-03-25T19:29:36.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:36.523+0000 7f0bc0396640 1 librados: shutdown 2024-03-25T19:29:36.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:41.169 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:41.880 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.174 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:41.885 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.194 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:09:41.905 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.220 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:41.874 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.229 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:41.940 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.235 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:41.889 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.241 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:41.952 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.268 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:09:41.921 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:41.279 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:09:41.991 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:29:43.342 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:43.458 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.458+0000 7fae63382640 10 monclient: get_monmap_and_config 2024-03-25T19:29:43.458 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.458+0000 7fae63382640 10 monclient: build_initial_monmap 2024-03-25T19:29:43.458 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.458+0000 7fae63382640 10 monclient: monmap: 2024-03-25T19:29:43.458 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:43.458 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:43.459793+0000 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:43.459793+0000 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:43.459 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:43.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.458+0000 7fae63382640 1 Processor -- start 2024-03-25T19:29:43.460 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.458+0000 7fae63382640 1 -- start start 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.458+0000 7fae63382640 10 monclient: init 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 10 monclient(hunting): picked mon.noname-b con 0x7fae5c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 10 monclient(hunting): picked mon.noname-a con 0x7fae5c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 10 monclient(hunting): picked mon.noname-c con 0x7fae5c14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c0af940 con 0x7fae5c0b1730 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c0b0a70 con 0x7fae5c14be90 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c0b0d50 con 0x7fae5c0b4d60 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:43.461 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.459+0000 7fae63382640 10 monclient(hunting): authenticate will time out at 6910.558608s 2024-03-25T19:29:43.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.474+0000 7fae6111e640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fae5c0b4d60 0x7fae5c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:33230/0 (socket says 172.21.15.60:33230) 2024-03-25T19:29:43.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.474+0000 7fae6111e640 1 -- 172.21.15.60:0/2929042062 learned_addr learned my addr 172.21.15.60:0/2929042062 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:43.482 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.482+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846988931 0 0) 0x7fae5c0af940 con 0x7fae5c0b1730 2024-03-25T19:29:43.482 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.482+0000 7fae53fff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:43.482 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.482+0000 7fae53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:43.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.482+0000 7fae53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:43.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.482+0000 7fae53fff640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:29:43.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.482+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae48003520 con 0x7fae5c0b1730 2024-03-25T19:29:43.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005908239 0 0) 0x7fae48003520 con 0x7fae5c0b1730 2024-03-25T19:29:43.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 >> v1:172.21.15.60:6790/0 conn(0x7fae5c0b4d60 legacy=0x7fae5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:43.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 >> v1:172.21.15.90:6789/0 conn(0x7fae5c14be90 legacy=0x7fae5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 1 monclient: found mon.noname-a 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c0b1030 con 0x7fae5c0b1730 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 10 monclient: _finish_auth 0 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fae4c001690 con 0x7fae5c0b1730 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae63382640 5 monclient: authenticate success, global_id 4650 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae63382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:43.490 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.488+0000 7fae53fff640 10 monclient: dump: 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:43.491 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.489+0000 7fae63382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:43.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae4c001860 con 0x7fae5c0b1730 2024-03-25T19:29:43.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:43.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae53fff640 1 -- 172.21.15.60:0/2929042062 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fae4c001cb0 con 0x7fae5c0b1730 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae63382640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae63382640 4 set_mon_vals no callback set 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae53fff640 10 monclient: dump: 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:43.517 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:43.518 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae63382640 10 monclient: shutdown 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae63382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.516+0000 7fae63382640 1 -- 172.21.15.60:0/2929042062 >> v1:172.21.15.60:6789/0 conn(0x7fae5c0b1730 legacy=0x7fae5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.517+0000 7fae63382640 1 -- 172.21.15.60:0/2929042062 shutdown_connections 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.517+0000 7fae63382640 1 -- 172.21.15.60:0/2929042062 >> 172.21.15.60:0/2929042062 conn(0x7fae5c0617a0 msgr2=0x7fae5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.517+0000 7fae63382640 1 -- 172.21.15.60:0/2929042062 shutdown_connections 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.517+0000 7fae63382640 1 -- 172.21.15.60:0/2929042062 wait complete. 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 10 monclient: build_initial_monmap 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 10 monclient: monmap: 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:43.520 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 1 librados: starting msgr at 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 1 librados: starting objecter 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 1 Processor -- start 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 1 -- start start 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 1 librados: setting wanted keys 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 1 librados: calling monclient init 2024-03-25T19:29:43.521 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.518+0000 7fae63382640 10 monclient: init 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 10 monclient(hunting): picked mon.noname-c con 0x7fae5c0b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 10 monclient(hunting): picked mon.noname-b con 0x7fae5c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 10 monclient(hunting): picked mon.noname-a con 0x7fae5c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c1431e0 con 0x7fae5c14be90 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c06f090 con 0x7fae5c0b4d60 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c141aa0 con 0x7fae5c0b1730 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:43.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.519+0000 7fae63382640 10 monclient(hunting): authenticate will time out at 6910.618463s 2024-03-25T19:29:43.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.529+0000 7fae6091d640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fae5c0b4d60 0x7fae5c141390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:50842/0 (socket says 172.21.15.60:50842) 2024-03-25T19:29:43.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.529+0000 7fae6091d640 1 -- 172.21.15.60:0/4246021492 learned_addr learned my addr 172.21.15.60:0/4246021492 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:43.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.533+0000 7fae51ffb640 10 client.?.objecter ms_handle_connect 0x7fae5c0b4d60 2024-03-25T19:29:43.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.533+0000 7fae51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:43.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.535+0000 7fae51ffb640 10 client.?.objecter ms_handle_connect 0x7fae5c14be90 2024-03-25T19:29:43.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.535+0000 7fae51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:43.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.540+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614466293 0 0) 0x7fae5c06f090 con 0x7fae5c0b4d60 2024-03-25T19:29:43.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.540+0000 7fae51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:43.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.540+0000 7fae51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:43.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.540+0000 7fae51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:43.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.540+0000 7fae51ffb640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:29:43.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.540+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae40004eb0 con 0x7fae5c0b4d60 2024-03-25T19:29:43.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.542+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326403226 0 0) 0x7fae5c1431e0 con 0x7fae5c14be90 2024-03-25T19:29:43.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.542+0000 7fae51ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:43.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.542+0000 7fae51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:43.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.542+0000 7fae51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:43.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.542+0000 7fae51ffb640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:29:43.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.542+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae5c06f090 con 0x7fae5c14be90 2024-03-25T19:29:43.554 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 669812317 0 0) 0x7fae5c06f090 con 0x7fae5c14be90 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae5c1431e0 con 0x7fae5c14be90 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fae54001a90 con 0x7fae5c14be90 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 10 monclient(hunting): dump: 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:43.555 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:43.556 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:43.556 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:43.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:43.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 10 client.?.objecter ms_handle_connect 0x7fae5c0b1730 2024-03-25T19:29:43.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.554+0000 7fae51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:43.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3452404364 0 0) 0x7fae5c1431e0 con 0x7fae5c14be90 2024-03-25T19:29:43.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 >> v1:172.21.15.60:6790/0 conn(0x7fae5c0b1730 legacy=0x7fae5c137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:43.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 >> v1:172.21.15.90:6789/0 conn(0x7fae5c0b4d60 legacy=0x7fae5c141390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:43.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:43.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 1 monclient: found mon.a 2024-03-25T19:29:43.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:43.566 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c141c30 con 0x7fae5c14be90 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 5 monclient: authenticate success, global_id 4653 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 10 monclient: _renew_subs 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c138670 con 0x7fae5c14be90 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 10 monclient: _renew_subs 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c138030 con 0x7fae5c14be90 2024-03-25T19:29:43.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.565+0000 7fae63382640 1 librados: init done 2024-03-25T19:29:43.568 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.566+0000 7fae63382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:43.568 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.566+0000 7fae63382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:43.568 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.566+0000 7fae63382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:43.568 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.566+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c0627b0 con 0x7fae5c14be90 2024-03-25T19:29:43.898 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.898+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae54001fa0 con 0x7fae5c14be90 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.898+0000 7fae51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.898+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fae54003970 con 0x7fae5c14be90 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.898+0000 7fae51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.898+0000 7fae51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.898+0000 7fae51ffb640 10 monclient: dump: 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:43.899 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:43.900 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:43.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.898+0000 7fae537fe640 4 set_mon_vals no callback set 2024-03-25T19:29:43.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.899+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fae54019940 con 0x7fae5c14be90 2024-03-25T19:29:43.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.899+0000 7fae6111e640 10 monclient: get_auth_request con 0x7fae4006dcb0 auth_method 0 2024-03-25T19:29:43.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.900+0000 7fae51ffb640 10 client.4653.objecter ms_handle_connect 0x7fae4006dcb0 2024-03-25T19:29:43.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fae540535b0 con 0x7fae5c14be90 2024-03-25T19:29:43.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 10 client.4653.objecter ms_dispatch 0x7fae5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:43.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:43.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:43.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T19:29:43.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae5404eef0 con 0x7fae5c14be90 2024-03-25T19:29:43.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:43.907 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:43.905+0000 7fae51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:44.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.137+0000 7fae63382640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:44.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.137+0000 7fae63382640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:44.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.137+0000 7fae63382640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:29:44.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.137+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae5c143530 con 0x7fae5c14be90 2024-03-25T19:29:44.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.183+0000 7fae51ffb640 1 -- 172.21.15.60:0/4246021492 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fae54050ff0 con 0x7fae5c14be90 2024-03-25T19:29:44.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.184+0000 7fae51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:44.185 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.184+0000 7fae51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:44.185 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:44.185 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:44.187 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 10 librados: watch_flush enter 2024-03-25T19:29:44.187 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 10 librados: watch_flush exit 2024-03-25T19:29:44.187 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 10 client.4653.objecter successfully canceled tick 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 >> v1:172.21.15.90:6800/1827638754 conn(0x7fae4006dcb0 legacy=0x7fae40070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 10 monclient: shutdown 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 >> v1:172.21.15.60:6789/0 conn(0x7fae5c14be90 legacy=0x7fae5c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 shutdown_connections 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 >> 172.21.15.60:0/4246021492 conn(0x7fae5c0617a0 msgr2=0x7fae5c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 shutdown_connections 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 1 -- 172.21.15.60:0/4246021492 wait complete. 2024-03-25T19:29:44.188 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:44.186+0000 7fae63382640 1 librados: shutdown 2024-03-25T19:29:44.199 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:51.006 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:51.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.121+0000 7f964fa4a640 10 monclient: get_monmap_and_config 2024-03-25T19:29:51.122 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.121+0000 7f964fa4a640 10 monclient: build_initial_monmap 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.121+0000 7f964fa4a640 10 monclient: monmap: 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:51.123405+0000 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:51.123405+0000 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.122+0000 7f964fa4a640 1 Processor -- start 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.122+0000 7f964fa4a640 1 -- start start 2024-03-25T19:29:51.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.122+0000 7f964fa4a640 10 monclient: init 2024-03-25T19:29:51.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:51.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f96480a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f964813f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f96481430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9648142520 con 0x7f964813f710 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480b28a0 con 0x7f96481430c0 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96480b2a30 con 0x7f96480a98f0 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:51.125 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.123+0000 7f964fa4a640 10 monclient(hunting): authenticate will time out at 6918.222429s 2024-03-25T19:29:51.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.128+0000 7f964dfe7640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f96481430c0 0x7f964814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:37198/0 (socket says 172.21.15.60:37198) 2024-03-25T19:29:51.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.128+0000 7f964dfe7640 1 -- 172.21.15.60:0/1730916472 learned_addr learned my addr 172.21.15.60:0/1730916472 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:51.142 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.141+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728665953 0 0) 0x7f96480b28a0 con 0x7f96481430c0 2024-03-25T19:29:51.142 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.141+0000 7f963ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:51.142 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.141+0000 7f963ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:51.142 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.141+0000 7f963ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:51.142 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.141+0000 7f963ffff640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:29:51.142 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.141+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9634003520 con 0x7f96481430c0 2024-03-25T19:29:51.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.151+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553739923 0 0) 0x7f96480b2a30 con 0x7f96480a98f0 2024-03-25T19:29:51.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.151+0000 7f963ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:29:51.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.151+0000 7f963ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:51.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.151+0000 7f963ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:51.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.151+0000 7f963ffff640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:29:51.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.151+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96480b28a0 con 0x7f96480a98f0 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3206631677 0 0) 0x7f9634003520 con 0x7f96481430c0 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 >> v1:172.21.15.60:6790/0 conn(0x7f96480a98f0 legacy=0x7f96480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 >> v1:172.21.15.60:6789/0 conn(0x7f964813f710 legacy=0x7f96481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 1 monclient: found mon.noname-c 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96480b44c0 con 0x7f96481430c0 2024-03-25T19:29:51.155 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 10 monclient: _finish_auth 0 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9640001a10 con 0x7f96481430c0 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f964fa4a640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f964fa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f963ffff640 10 monclient: dump: 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:51.156 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:51.157 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:51.157 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:51.157 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:51.157 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:51.157 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.154+0000 7f964fa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:51.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9640001b70 con 0x7f96481430c0 2024-03-25T19:29:51.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f963ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:51.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f963ffff640 1 -- 172.21.15.60:0/1730916472 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9640001f80 con 0x7f96481430c0 2024-03-25T19:29:51.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f964fa4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:51.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f964fa4a640 4 set_mon_vals no callback set 2024-03-25T19:29:51.169 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f963ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:51.169 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f963ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:51.169 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f963ffff640 10 monclient: dump: 2024-03-25T19:29:51.169 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:51.170 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:51.172 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f964fa4a640 10 monclient: shutdown 2024-03-25T19:29:51.172 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f964fa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:51.172 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.168+0000 7f964fa4a640 1 -- 172.21.15.60:0/1730916472 >> v1:172.21.15.90:6789/0 conn(0x7f96481430c0 legacy=0x7f964814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.172 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.169+0000 7f964fa4a640 1 -- 172.21.15.60:0/1730916472 shutdown_connections 2024-03-25T19:29:51.172 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.169+0000 7f964fa4a640 1 -- 172.21.15.60:0/1730916472 >> 172.21.15.60:0/1730916472 conn(0x7f96480b7ea0 msgr2=0x7f96480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.169+0000 7f964fa4a640 1 -- 172.21.15.60:0/1730916472 shutdown_connections 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.169+0000 7f964fa4a640 1 -- 172.21.15.60:0/1730916472 wait complete. 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 10 monclient: build_initial_monmap 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 10 monclient: monmap: 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:51.173 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 1 librados: starting msgr at 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 1 librados: starting objecter 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 1 Processor -- start 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 1 -- start start 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 1 librados: setting wanted keys 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 1 librados: calling monclient init 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.170+0000 7f964fa4a640 10 monclient: init 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f96480a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f964813f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f96481430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:51.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9648137c60 con 0x7f96481430c0 2024-03-25T19:29:51.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9648139160 con 0x7f96480a98f0 2024-03-25T19:29:51.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96481380f0 con 0x7f964813f710 2024-03-25T19:29:51.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:51.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.171+0000 7f964fa4a640 10 monclient(hunting): authenticate will time out at 6918.270352s 2024-03-25T19:29:51.176 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.176+0000 7f964d7e6640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f96480a98f0 0x7f96481dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:37236/0 (socket says 172.21.15.60:37236) 2024-03-25T19:29:51.176 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.176+0000 7f964d7e6640 1 -- 172.21.15.60:0/1679450185 learned_addr learned my addr 172.21.15.60:0/1679450185 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:51.177 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.176+0000 7f963dffb640 10 client.?.objecter ms_handle_connect 0x7f96480a98f0 2024-03-25T19:29:51.177 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.176+0000 7f963dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:51.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.183+0000 7f963dffb640 10 client.?.objecter ms_handle_connect 0x7f964813f710 2024-03-25T19:29:51.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.183+0000 7f963dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:51.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.185+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543125818 0 0) 0x7f9648139160 con 0x7f96480a98f0 2024-03-25T19:29:51.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.186+0000 7f963dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:51.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.186+0000 7f963dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:51.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.186+0000 7f963dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:51.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.186+0000 7f963dffb640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:29:51.186 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.186+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f962c004eb0 con 0x7f96480a98f0 2024-03-25T19:29:51.189 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.189+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542356102 0 0) 0x7f96481380f0 con 0x7f964813f710 2024-03-25T19:29:51.189 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.189+0000 7f963dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:29:51.189 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.189+0000 7f963dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:51.189 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.189+0000 7f963dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:51.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.189+0000 7f963dffb640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:29:51.190 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.189+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9648139160 con 0x7f964813f710 2024-03-25T19:29:51.199 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1454771203 0 0) 0x7f9648139160 con 0x7f964813f710 2024-03-25T19:29:51.199 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96481380f0 con 0x7f964813f710 2024-03-25T19:29:51.199 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9638001930 con 0x7f964813f710 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 10 monclient(hunting): dump: 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:51.200 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:51.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 10 client.?.objecter ms_handle_connect 0x7f96481430c0 2024-03-25T19:29:51.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.199+0000 7f963dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:51.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.201+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3849701829 0 0) 0x7f962c004eb0 con 0x7f96480a98f0 2024-03-25T19:29:51.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.201+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9648139160 con 0x7f96480a98f0 2024-03-25T19:29:51.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.201+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9644001a10 con 0x7f96480a98f0 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.201+0000 7f963dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.201+0000 7f963dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.201+0000 7f963dffb640 10 monclient(hunting): dump: 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:51.202 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:51.203 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:51.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.208+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1030111868 0 0) 0x7f96481380f0 con 0x7f964813f710 2024-03-25T19:29:51.209 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.208+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 >> v1:172.21.15.90:6789/0 conn(0x7f96480a98f0 legacy=0x7f96481dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.208+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 >> v1:172.21.15.60:6789/0 conn(0x7f96481430c0 legacy=0x7f96481e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 1 monclient: found mon.c 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96481397f0 con 0x7f964813f710 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 10 monclient: _finish_auth 0 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f963dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 5 monclient: authenticate success, global_id 4640 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 10 monclient: _renew_subs 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:51.210 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96481db280 con 0x7f964813f710 2024-03-25T19:29:51.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:51.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 10 monclient: _renew_subs 2024-03-25T19:29:51.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:51.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9648133780 con 0x7f964813f710 2024-03-25T19:29:51.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.209+0000 7f964fa4a640 1 librados: init done 2024-03-25T19:29:51.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.210+0000 7f964fa4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:51.212 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.210+0000 7f964fa4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:51.212 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.210+0000 7f964fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:51.212 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.210+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9648139a40 con 0x7f964813f710 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.532+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9638003610 con 0x7f964813f710 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.532+0000 7f963dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.532+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9638003960 con 0x7f964813f710 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.532+0000 7f963dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.532+0000 7f963dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.532+0000 7f963dffb640 10 monclient: dump: 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:51.533 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:51.534 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:51.534 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:51.534 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:51.534 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:51.534 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:51.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.532+0000 7f963f7fe640 4 set_mon_vals no callback set 2024-03-25T19:29:51.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.533+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f9638003b80 con 0x7f964813f710 2024-03-25T19:29:51.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.534+0000 7f964d7e6640 10 monclient: get_auth_request con 0x7f962c06ded0 auth_method 0 2024-03-25T19:29:51.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.534+0000 7f963dffb640 10 client.4640.objecter ms_handle_connect 0x7f962c06ded0 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f9638053600 con 0x7f964813f710 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 10 client.4640.objecter ms_dispatch 0x7f9648000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f963804f110 con 0x7f964813f710 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:51.542 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.541+0000 7f963dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:51.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.776+0000 7f964fa4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:51.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.776+0000 7f964fa4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:51.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.776+0000 7f964fa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:51.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.776+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9648133ad0 con 0x7f964813f710 2024-03-25T19:29:51.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.803+0000 7f963dffb640 1 -- 172.21.15.60:0/1679450185 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f9638051040 con 0x7f964813f710 2024-03-25T19:29:51.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.803+0000 7f963dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:51.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.803+0000 7f963dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:51.804 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:51.804 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:51.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 10 librados: watch_flush enter 2024-03-25T19:29:51.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 10 librados: watch_flush exit 2024-03-25T19:29:51.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-25T19:29:51.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 10 client.4640.objecter successfully canceled tick 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 >> v1:172.21.15.90:6800/1827638754 conn(0x7f962c06ded0 legacy=0x7f962c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 10 monclient: shutdown 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 >> v1:172.21.15.60:6790/0 conn(0x7f964813f710 legacy=0x7f9648133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 shutdown_connections 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.805+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 >> 172.21.15.60:0/1679450185 conn(0x7f96480b7ea0 msgr2=0x7f964813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.806+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 shutdown_connections 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.806+0000 7f964fa4a640 1 -- 172.21.15.60:0/1679450185 wait complete. 2024-03-25T19:29:51.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:51.806+0000 7f964fa4a640 1 librados: shutdown 2024-03-25T19:29:51.817 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:29:58.624 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:29:58.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.743+0000 7fac9bf0e640 10 monclient: get_monmap_and_config 2024-03-25T19:29:58.743 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.743+0000 7fac9bf0e640 10 monclient: build_initial_monmap 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.743+0000 7fac9bf0e640 10 monclient: monmap: 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:29:58.744784+0000 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:29:58.744784+0000 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:29:58.744 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:58.745 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.743+0000 7fac9bf0e640 1 Processor -- start 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.744+0000 7fac9bf0e640 1 -- start start 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.744+0000 7fac9bf0e640 10 monclient: init 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.744+0000 7fac9bf0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 10 monclient(hunting): picked mon.noname-b con 0x7fac940b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 10 monclient(hunting): picked mon.noname-c con 0x7fac940b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 10 monclient(hunting): picked mon.noname-a con 0x7fac9414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940af940 con 0x7fac9414be90 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940b0a70 con 0x7fac940b1730 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940b0d50 con 0x7fac940b4d60 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:58.746 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.745+0000 7fac9bf0e640 10 monclient(hunting): authenticate will time out at 6925.844179s 2024-03-25T19:29:58.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.750+0000 7fac9a4ab640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fac9414be90 0x7fac9414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:57960/0 (socket says 172.21.15.60:57960) 2024-03-25T19:29:58.750 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.750+0000 7fac9a4ab640 1 -- 172.21.15.60:0/3690096456 learned_addr learned my addr 172.21.15.60:0/3690096456 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:58.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.757+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600003008 0 0) 0x7fac940af940 con 0x7fac9414be90 2024-03-25T19:29:58.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.757+0000 7fac98ca8640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.757+0000 7fac98ca8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.757+0000 7fac98ca8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.757+0000 7fac98ca8640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:29:58.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.757+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac80003520 con 0x7fac9414be90 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 875836304 0 0) 0x7fac80003520 con 0x7fac9414be90 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 >> v1:172.21.15.60:6790/0 conn(0x7fac940b4d60 legacy=0x7fac9413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 >> v1:172.21.15.90:6789/0 conn(0x7fac940b1730 legacy=0x7fac940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 10 monclient: _finish_hunting 0 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 1 monclient: found mon.noname-a 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940b1030 con 0x7fac9414be90 2024-03-25T19:29:58.768 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 10 monclient: _finish_auth 0 2024-03-25T19:29:58.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 10 monclient: _check_auth_tickets 2024-03-25T19:29:58.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac98ca8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:58.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac9bf0e640 5 monclient: authenticate success, global_id 4662 2024-03-25T19:29:58.769 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.767+0000 7fac9bf0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:58.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fac8c001a10 con 0x7fac9414be90 2024-03-25T19:29:58.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 10 monclient: dump: 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac8c001ca0 con 0x7fac9414be90 2024-03-25T19:29:58.773 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac9bf0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 1 -- 172.21.15.60:0/3690096456 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fac8c001fb0 con 0x7fac9414be90 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac9bf0e640 10 monclient: get_monmap_and_config success 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac9bf0e640 4 set_mon_vals no callback set 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac98ca8640 10 monclient: dump: 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.774 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.775 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:58.775 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:58.775 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:58.775 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:58.775 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:58.776 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac9bf0e640 10 monclient: shutdown 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac9bf0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.772+0000 7fac9bf0e640 1 -- 172.21.15.60:0/3690096456 >> v1:172.21.15.60:6789/0 conn(0x7fac9414be90 legacy=0x7fac9414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.773+0000 7fac9bf0e640 1 -- 172.21.15.60:0/3690096456 shutdown_connections 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.773+0000 7fac9bf0e640 1 -- 172.21.15.60:0/3690096456 >> 172.21.15.60:0/3690096456 conn(0x7fac940617a0 msgr2=0x7fac94061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.773+0000 7fac9bf0e640 1 -- 172.21.15.60:0/3690096456 shutdown_connections 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.773+0000 7fac9bf0e640 1 -- 172.21.15.60:0/3690096456 wait complete. 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.773+0000 7fac9bf0e640 10 monclient: build_initial_monmap 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.773+0000 7fac9bf0e640 10 monclient: monmap: 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:29:58.777 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 librados: starting msgr at 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 librados: starting objecter 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 Processor -- start 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 -- start start 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 librados: setting wanted keys 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 librados: calling monclient init 2024-03-25T19:29:58.778 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 10 monclient: init 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 10 monclient: _reopen_session rank -1 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 10 monclient(hunting): picked mon.noname-c con 0x7fac940b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 10 monclient(hunting): picked mon.noname-a con 0x7fac940b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 10 monclient(hunting): picked mon.noname-b con 0x7fac9414be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac941379e0 con 0x7fac940b4d60 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.774+0000 7fac9bf0e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac94138530 con 0x7fac9414be90 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.775+0000 7fac9bf0e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac941381c0 con 0x7fac940b1730 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.775+0000 7fac9bf0e640 10 monclient(hunting): _renew_subs 2024-03-25T19:29:58.779 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.775+0000 7fac9bf0e640 10 monclient(hunting): authenticate will time out at 6925.873898s 2024-03-25T19:29:58.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.783+0000 7fac9a4ab640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fac9414be90 0x7fac941e2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:58154/0 (socket says 172.21.15.60:58154) 2024-03-25T19:29:58.783 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.783+0000 7fac9a4ab640 1 -- 172.21.15.60:0/2646881884 learned_addr learned my addr 172.21.15.60:0/2646881884 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:29:58.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.784+0000 7fac8a7fc640 10 client.?.objecter ms_handle_connect 0x7fac9414be90 2024-03-25T19:29:58.784 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.784+0000 7fac8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.787+0000 7fac8a7fc640 10 client.?.objecter ms_handle_connect 0x7fac940b4d60 2024-03-25T19:29:58.787 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.787+0000 7fac8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.789+0000 7fac8a7fc640 10 client.?.objecter ms_handle_connect 0x7fac940b1730 2024-03-25T19:29:58.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.789+0000 7fac8a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:29:58.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.790+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425113118 0 0) 0x7fac94138530 con 0x7fac9414be90 2024-03-25T19:29:58.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.790+0000 7fac8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.790+0000 7fac8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.790+0000 7fac8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.790 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.790+0000 7fac8a7fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:29:58.791 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.790+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac78004eb0 con 0x7fac9414be90 2024-03-25T19:29:58.802 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411809927 0 0) 0x7fac941379e0 con 0x7fac940b4d60 2024-03-25T19:29:58.802 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac94138530 con 0x7fac940b4d60 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947392317 0 0) 0x7fac941381c0 con 0x7fac940b1730 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:29:58.803 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:29:58.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:29:58.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.802+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac941379e0 con 0x7fac940b1730 2024-03-25T19:29:58.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.814+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203731335 0 0) 0x7fac941379e0 con 0x7fac940b1730 2024-03-25T19:29:58.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.814+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac941381c0 con 0x7fac940b1730 2024-03-25T19:29:58.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.814+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fac90001830 con 0x7fac940b1730 2024-03-25T19:29:58.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.814+0000 7fac8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.814+0000 7fac8a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:29:58.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.814+0000 7fac8a7fc640 10 monclient(hunting): dump: 2024-03-25T19:29:58.815 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:58.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.814+0000 7fac8a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.816+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1116185677 0 0) 0x7fac78004eb0 con 0x7fac9414be90 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.816+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac941379e0 con 0x7fac9414be90 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.816+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fac8c0019d0 con 0x7fac9414be90 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.816+0000 7fac8a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.816+0000 7fac8a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.816+0000 7fac8a7fc640 10 monclient(hunting): dump: 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:58.817 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.818 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:58.818 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:58.818 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:58.818 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:58.818 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:58.818 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:58.818 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 980076442 0 0) 0x7fac941381c0 con 0x7fac940b1730 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 >> v1:172.21.15.90:6789/0 conn(0x7fac9414be90 legacy=0x7fac941e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 >> v1:172.21.15.60:6789/0 conn(0x7fac940b4d60 legacy=0x7fac941d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 1 monclient: found mon.c 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac941355e0 con 0x7fac940b1730 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 5 monclient: authenticate success, global_id 4646 2024-03-25T19:29:58.830 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 10 monclient: _renew_subs 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9413eb10 con 0x7fac940b1730 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 10 monclient: _renew_subs 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac941d7ca0 con 0x7fac940b1730 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.828+0000 7fac9bf0e640 1 librados: init done 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.829+0000 7fac9bf0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.829+0000 7fac9bf0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.829+0000 7fac9bf0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:58.831 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:58.829+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac941386c0 con 0x7fac940b1730 2024-03-25T19:29:59.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac90001d40 con 0x7fac940b1730 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fac90003790 con 0x7fac940b1730 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8a7fc640 10 monclient: dump: 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:29:59.139 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:29:59.140 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:29:59.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8bfff640 4 set_mon_vals no callback set 2024-03-25T19:29:59.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.138+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fac900039b0 con 0x7fac940b1730 2024-03-25T19:29:59.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.139+0000 7fac994a9640 10 monclient: get_auth_request con 0x7fac7806df70 auth_method 0 2024-03-25T19:29:59.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.139+0000 7fac8a7fc640 10 client.4646.objecter ms_handle_connect 0x7fac7806df70 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fac900535b0 con 0x7fac940b1730 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 10 client.4646.objecter ms_dispatch 0x7fac94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac9004eef0 con 0x7fac940b1730 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:29:59.146 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.145+0000 7fac8a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:29:59.405 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.404+0000 7fac9bf0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:59.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.404+0000 7fac9bf0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:59.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.404+0000 7fac9bf0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:29:59.406 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.404+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac94138350 con 0x7fac940b1730 2024-03-25T19:29:59.444 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.444+0000 7fac8a7fc640 1 -- 172.21.15.60:0/2646881884 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fac90050ff0 con 0x7fac940b1730 2024-03-25T19:29:59.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.444+0000 7fac8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:29:59.445 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.444+0000 7fac8a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:29:59.445 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:29:59.445 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:29:59.447 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.446+0000 7fac9bf0e640 10 librados: watch_flush enter 2024-03-25T19:29:59.447 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.446+0000 7fac9bf0e640 10 librados: watch_flush exit 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.446+0000 7fac9bf0e640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.446+0000 7fac9bf0e640 10 client.4646.objecter successfully canceled tick 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.446+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 >> v1:172.21.15.90:6800/1827638754 conn(0x7fac7806df70 legacy=0x7fac78070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 10 monclient: shutdown 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 >> v1:172.21.15.60:6790/0 conn(0x7fac940b1730 legacy=0x7fac94134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 shutdown_connections 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 >> 172.21.15.60:0/2646881884 conn(0x7fac940617a0 msgr2=0x7fac940b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 shutdown_connections 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 1 -- 172.21.15.60:0/2646881884 wait complete. 2024-03-25T19:29:59.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:29:59.447+0000 7fac9bf0e640 1 librados: shutdown 2024-03-25T19:29:59.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:01.184 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:01.895 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.228 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:01.881 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.229 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:01.883 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.238 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:01.949 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.261 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:01.915 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.268 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:01.979 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.305 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:02.016 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.310 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:02.021 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:01.342 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:02.053 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:06.266 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:06.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.381+0000 7f2e2b932640 10 monclient: get_monmap_and_config 2024-03-25T19:30:06.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.381+0000 7f2e2b932640 10 monclient: build_initial_monmap 2024-03-25T19:30:06.381 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.381+0000 7f2e2b932640 10 monclient: monmap: 2024-03-25T19:30:06.381 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:06.381 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:30:06.382776+0000 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:30:06.382776+0000 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:30:06.382 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.383 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.381+0000 7f2e2b932640 1 Processor -- start 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.381+0000 7f2e2b932640 1 -- start start 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.381+0000 7f2e2b932640 10 monclient: init 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.382+0000 7f2e2b932640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.382+0000 7f2e2b932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.382+0000 7f2e2b932640 10 monclient(hunting): picked mon.noname-a con 0x7f2e240b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.382+0000 7f2e2b932640 10 monclient(hunting): picked mon.noname-c con 0x7f2e240b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.383+0000 7f2e2b932640 10 monclient(hunting): picked mon.noname-b con 0x7f2e2414be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.383+0000 7f2e2b932640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240af940 con 0x7f2e240b4d60 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.383+0000 7f2e2b932640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240b0a70 con 0x7f2e240b1730 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.383+0000 7f2e2b932640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240b0d50 con 0x7f2e2414be90 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.383+0000 7f2e2b932640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:06.384 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.383+0000 7f2e2b932640 10 monclient(hunting): authenticate will time out at 6933.482136s 2024-03-25T19:30:06.391 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.391+0000 7f2e29ecf640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f2e2414be90 0x7f2e2414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:48428/0 (socket says 172.21.15.60:48428) 2024-03-25T19:30:06.391 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.391+0000 7f2e29ecf640 1 -- 172.21.15.60:0/2878290145 learned_addr learned my addr 172.21.15.60:0/2878290145 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:06.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.409+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716359771 0 0) 0x7f2e240b0a70 con 0x7f2e240b1730 2024-03-25T19:30:06.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.409+0000 7f2e1bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:06.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.409+0000 7f2e1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:06.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.409+0000 7f2e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:06.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.409+0000 7f2e1bfff640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:30:06.410 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.409+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e10003520 con 0x7f2e240b1730 2024-03-25T19:30:06.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.421+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4018930512 0 0) 0x7f2e240af940 con 0x7f2e240b4d60 2024-03-25T19:30:06.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.421+0000 7f2e1bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:06.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.421+0000 7f2e1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:06.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.421+0000 7f2e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:06.421 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.421+0000 7f2e1bfff640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:30:06.422 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.421+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e240b0a70 con 0x7f2e240b4d60 2024-03-25T19:30:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.434+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738983706 0 0) 0x7f2e240b0d50 con 0x7f2e2414be90 2024-03-25T19:30:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.434+0000 7f2e1bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.434+0000 7f2e1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.434+0000 7f2e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.434+0000 7f2e1bfff640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:30:06.434 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.434+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e240af940 con 0x7f2e2414be90 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913536332 0 0) 0x7f2e10003520 con 0x7f2e240b1730 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 >> v1:172.21.15.60:6790/0 conn(0x7f2e2414be90 legacy=0x7f2e2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 >> v1:172.21.15.60:6789/0 conn(0x7f2e240b4d60 legacy=0x7f2e2413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 1 monclient: found mon.noname-c 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e240b1030 con 0x7f2e240b1730 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 10 monclient: _finish_auth 0 2024-03-25T19:30:06.442 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e14001690 con 0x7f2e240b1730 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e2b932640 5 monclient: authenticate success, global_id 4654 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e2b932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e1bfff640 10 monclient: dump: 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:06.443 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:06.444 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:06.444 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.444 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.441+0000 7f2e2b932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:06.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e140017f0 con 0x7f2e240b1730 2024-03-25T19:30:06.467 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e1bfff640 1 -- 172.21.15.60:0/2878290145 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e14001c00 con 0x7f2e240b1730 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e2b932640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e2b932640 4 set_mon_vals no callback set 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e1bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.467+0000 7f2e1bfff640 10 monclient: dump: 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:06.468 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.468+0000 7f2e2b932640 10 monclient: shutdown 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.468+0000 7f2e2b932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.468+0000 7f2e2b932640 1 -- 172.21.15.60:0/2878290145 >> v1:172.21.15.90:6789/0 conn(0x7f2e240b1730 legacy=0x7f2e240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.468+0000 7f2e2b932640 1 -- 172.21.15.60:0/2878290145 shutdown_connections 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.468+0000 7f2e2b932640 1 -- 172.21.15.60:0/2878290145 >> 172.21.15.60:0/2878290145 conn(0x7f2e240617a0 msgr2=0x7f2e24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.468+0000 7f2e2b932640 1 -- 172.21.15.60:0/2878290145 shutdown_connections 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.468+0000 7f2e2b932640 1 -- 172.21.15.60:0/2878290145 wait complete. 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 10 monclient: build_initial_monmap 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 10 monclient: monmap: 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:30:06.471 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 1 librados: starting msgr at 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 1 librados: starting objecter 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 1 Processor -- start 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 1 -- start start 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 1 librados: setting wanted keys 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 1 librados: calling monclient init 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.469+0000 7f2e2b932640 10 monclient: init 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:06.472 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 10 monclient(hunting): picked mon.noname-b con 0x7f2e240b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:06.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 10 monclient(hunting): picked mon.noname-c con 0x7f2e240b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:06.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 10 monclient(hunting): picked mon.noname-a con 0x7f2e2414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:06.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2413bd10 con 0x7f2e2414be90 2024-03-25T19:30:06.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e24141c30 con 0x7f2e240b1730 2024-03-25T19:30:06.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e24141f10 con 0x7f2e240b4d60 2024-03-25T19:30:06.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:06.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.470+0000 7f2e2b932640 10 monclient(hunting): authenticate will time out at 6933.569304s 2024-03-25T19:30:06.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.479+0000 7f2e29ecf640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f2e2414be90 0x7f2e241e35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47464/0 (socket says 172.21.15.60:47464) 2024-03-25T19:30:06.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.479+0000 7f2e29ecf640 1 -- 172.21.15.60:0/202457574 learned_addr learned my addr 172.21.15.60:0/202457574 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:06.480 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.480+0000 7f2e19ffb640 10 client.?.objecter ms_handle_connect 0x7f2e2414be90 2024-03-25T19:30:06.481 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.480+0000 7f2e19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:06.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.483+0000 7f2e19ffb640 10 client.?.objecter ms_handle_connect 0x7f2e240b4d60 2024-03-25T19:30:06.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.483+0000 7f2e19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:06.486 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.485+0000 7f2e19ffb640 10 client.?.objecter ms_handle_connect 0x7f2e240b1730 2024-03-25T19:30:06.486 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.485+0000 7f2e19ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:06.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.497+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772292703 0 0) 0x7f2e2413bd10 con 0x7f2e2414be90 2024-03-25T19:30:06.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.497+0000 7f2e19ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:06.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.497+0000 7f2e19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:06.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.497+0000 7f2e19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:06.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.497+0000 7f2e19ffb640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:30:06.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.497+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e08004eb0 con 0x7f2e2414be90 2024-03-25T19:30:06.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.499+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951168738 0 0) 0x7f2e24141f10 con 0x7f2e240b4d60 2024-03-25T19:30:06.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.499+0000 7f2e19ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:06.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.499+0000 7f2e19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:06.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.499+0000 7f2e19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:06.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.499+0000 7f2e19ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:30:06.500 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.499+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2413bd10 con 0x7f2e240b4d60 2024-03-25T19:30:06.503 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.502+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832154949 0 0) 0x7f2e24141c30 con 0x7f2e240b1730 2024-03-25T19:30:06.503 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.502+0000 7f2e19ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:06.503 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.502+0000 7f2e19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:06.503 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.502+0000 7f2e19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:06.503 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.502+0000 7f2e19ffb640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:30:06.503 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.502+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e24141f10 con 0x7f2e240b1730 2024-03-25T19:30:06.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.522+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2117162853 0 0) 0x7f2e2413bd10 con 0x7f2e240b4d60 2024-03-25T19:30:06.522 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.522+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e24141c30 con 0x7f2e240b4d60 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.522+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e14001b00 con 0x7f2e240b4d60 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.522+0000 7f2e19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.522+0000 7f2e19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.522+0000 7f2e19ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:06.523 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.524 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.522+0000 7f2e19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.527+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357531664 0 0) 0x7f2e08004eb0 con 0x7f2e2414be90 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.527+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e2413bd10 con 0x7f2e2414be90 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.527+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e1c001bd0 con 0x7f2e2414be90 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.527+0000 7f2e19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.527+0000 7f2e19ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.527+0000 7f2e19ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:06.528 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:06.529 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:06.529 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.534+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2844258997 0 0) 0x7f2e24141f10 con 0x7f2e240b1730 2024-03-25T19:30:06.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.534+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e08004eb0 con 0x7f2e240b1730 2024-03-25T19:30:06.534 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.534+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e200018f0 con 0x7f2e240b1730 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.534+0000 7f2e19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.534+0000 7f2e19ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.534+0000 7f2e19ffb640 10 monclient(hunting): dump: 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:06.535 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3758198346 0 0) 0x7f2e2413bd10 con 0x7f2e2414be90 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 >> v1:172.21.15.60:6790/0 conn(0x7f2e240b4d60 legacy=0x7f2e24142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 >> v1:172.21.15.90:6789/0 conn(0x7f2e240b1730 legacy=0x7f2e241341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 1 monclient: found mon.a 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:06.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24140810 con 0x7f2e2414be90 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 5 monclient: authenticate success, global_id 4671 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 10 monclient: _renew_subs 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24140d30 con 0x7f2e2414be90 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 10 client.4671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 10 monclient: _renew_subs 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e24140560 con 0x7f2e2414be90 2024-03-25T19:30:06.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.607+0000 7f2e2b932640 1 librados: init done 2024-03-25T19:30:06.610 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.608+0000 7f2e2b932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:06.610 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.608+0000 7f2e2b932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:06.610 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.608+0000 7f2e2b932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:06.610 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.608+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e24140f20 con 0x7f2e2414be90 2024-03-25T19:30:06.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.920+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e1c002080 con 0x7f2e2414be90 2024-03-25T19:30:06.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.920+0000 7f2e19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:06.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.920+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2e1c004d00 con 0x7f2e2414be90 2024-03-25T19:30:06.921 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.920+0000 7f2e19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.920+0000 7f2e19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.920+0000 7f2e19ffb640 10 monclient: dump: 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:06.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.920+0000 7f2e1b7fe640 4 set_mon_vals no callback set 2024-03-25T19:30:06.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.921+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2e1c004f20 con 0x7f2e2414be90 2024-03-25T19:30:06.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.922+0000 7f2e296ce640 10 monclient: get_auth_request con 0x7f2e0806e080 auth_method 0 2024-03-25T19:30:06.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.922+0000 7f2e19ffb640 10 client.4671.objecter ms_handle_connect 0x7f2e0806e080 2024-03-25T19:30:06.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7f2e1c054a70 con 0x7f2e2414be90 2024-03-25T19:30:06.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 10 client.4671.objecter ms_dispatch 0x7f2e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:06.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 3 client.4671.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:06.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 3 client.4671.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:06.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 20 client.4671.objecter dump_active .. 0 homeless 2024-03-25T19:30:06.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e1c0503b0 con 0x7f2e2414be90 2024-03-25T19:30:06.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:06.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:06.929+0000 7f2e19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:07.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.191+0000 7f2e2b932640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:07.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.191+0000 7f2e2b932640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:07.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.191+0000 7f2e2b932640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:07.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.191+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e240627b0 con 0x7f2e2414be90 2024-03-25T19:30:07.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.225+0000 7f2e19ffb640 1 -- 172.21.15.60:0/202457574 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7f2e1c0524b0 con 0x7f2e2414be90 2024-03-25T19:30:07.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.225+0000 7f2e19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:07.226 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.225+0000 7f2e19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:07.226 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:30:07.226 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:07.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.227+0000 7f2e2b932640 10 librados: watch_flush enter 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.227+0000 7f2e2b932640 10 librados: watch_flush exit 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.227+0000 7f2e2b932640 20 client.4671.objecter shutdown clearing up homeless session... 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.227+0000 7f2e2b932640 10 client.4671.objecter successfully canceled tick 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 >> v1:172.21.15.90:6800/1827638754 conn(0x7f2e0806e080 legacy=0x7f2e08070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 10 monclient: shutdown 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 >> v1:172.21.15.60:6789/0 conn(0x7f2e2414be90 legacy=0x7f2e241e35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 shutdown_connections 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 >> 172.21.15.60:0/202457574 conn(0x7f2e240617a0 msgr2=0x7f2e240b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 shutdown_connections 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 1 -- 172.21.15.60:0/202457574 wait complete. 2024-03-25T19:30:07.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:07.228+0000 7f2e2b932640 1 librados: shutdown 2024-03-25T19:30:07.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:14.047 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:14.162 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.163+0000 7fbd6dee2640 10 monclient: get_monmap_and_config 2024-03-25T19:30:14.162 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.163+0000 7fbd6dee2640 10 monclient: build_initial_monmap 2024-03-25T19:30:14.162 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.163+0000 7fbd6dee2640 10 monclient: monmap: 2024-03-25T19:30:14.162 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:14.162 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:30:14.163751+0000 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:30:14.163751+0000 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:30:14.163 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:14.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.163+0000 7fbd6dee2640 1 Processor -- start 2024-03-25T19:30:14.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.163+0000 7fbd6dee2640 1 -- start start 2024-03-25T19:30:14.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.163+0000 7fbd6dee2640 10 monclient: init 2024-03-25T19:30:14.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:14.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:14.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 10 monclient(hunting): picked mon.noname-c con 0x7fbd680b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:14.164 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 10 monclient(hunting): picked mon.noname-a con 0x7fbd680b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:14.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 10 monclient(hunting): picked mon.noname-b con 0x7fbd6814be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:14.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd680af940 con 0x7fbd680b1730 2024-03-25T19:30:14.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd680b0a70 con 0x7fbd680b4d60 2024-03-25T19:30:14.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd680b0d50 con 0x7fbd6814be90 2024-03-25T19:30:14.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:14.165 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.164+0000 7fbd6dee2640 10 monclient(hunting): authenticate will time out at 6941.262507s 2024-03-25T19:30:14.176 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.176+0000 7fbd66ffd640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7fbd680b1730 0x7fbd680b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:52236/0 (socket says 172.21.15.60:52236) 2024-03-25T19:30:14.176 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.176+0000 7fbd66ffd640 1 -- 172.21.15.60:0/3740861703 learned_addr learned my addr 172.21.15.60:0/3740861703 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:14.185 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.186+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087606396 0 0) 0x7fbd680af940 con 0x7fbd680b1730 2024-03-25T19:30:14.185 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.186+0000 7fbd667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:14.185 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.186+0000 7fbd667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:14.185 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.186+0000 7fbd667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:14.185 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.186+0000 7fbd667fc640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:30:14.185 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.186+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd50003520 con 0x7fbd680b1730 2024-03-25T19:30:14.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.193+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957363362 0 0) 0x7fbd680b0a70 con 0x7fbd680b4d60 2024-03-25T19:30:14.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.193+0000 7fbd667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:14.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.193+0000 7fbd667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:14.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.193+0000 7fbd667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:14.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.193+0000 7fbd667fc640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:30:14.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.193+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd680af940 con 0x7fbd680b4d60 2024-03-25T19:30:14.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.202+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338345468 0 0) 0x7fbd680b0d50 con 0x7fbd6814be90 2024-03-25T19:30:14.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.202+0000 7fbd667fc640 10 monclient(hunting): _negotiate 2024-03-25T19:30:14.201 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.202+0000 7fbd667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:14.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.202+0000 7fbd667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:14.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.202+0000 7fbd667fc640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:30:14.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.202+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd680b0a70 con 0x7fbd6814be90 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836430637 0 0) 0x7fbd50003520 con 0x7fbd680b1730 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 >> v1:172.21.15.60:6790/0 conn(0x7fbd6814be90 legacy=0x7fbd6814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 >> v1:172.21.15.90:6789/0 conn(0x7fbd680b4d60 legacy=0x7fbd6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 10 monclient: _finish_hunting 0 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 1 monclient: found mon.noname-a 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd680b1030 con 0x7fbd680b1730 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 10 monclient: _finish_auth 0 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 10 monclient: _check_auth_tickets 2024-03-25T19:30:14.214 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd5c001690 con 0x7fbd680b1730 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd6dee2640 5 monclient: authenticate success, global_id 4674 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd6dee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd667fc640 10 monclient: dump: 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:14.215 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:14.216 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.214+0000 7fbd6dee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd5c0017f0 con 0x7fbd680b1730 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd667fc640 1 -- 172.21.15.60:0/3740861703 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd5c001c00 con 0x7fbd680b1730 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd6dee2640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd6dee2640 4 set_mon_vals no callback set 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.219+0000 7fbd667fc640 10 monclient: dump: 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.219 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:14.220 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:14.220 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:14.220 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:14.220 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:14.220 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:14.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.220+0000 7fbd6dee2640 10 monclient: shutdown 2024-03-25T19:30:14.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.220+0000 7fbd6dee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:14.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.220+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3740861703 >> v1:172.21.15.60:6789/0 conn(0x7fbd680b1730 legacy=0x7fbd680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:14.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.220+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3740861703 shutdown_connections 2024-03-25T19:30:14.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.220+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3740861703 >> 172.21.15.60:0/3740861703 conn(0x7fbd680617a0 msgr2=0x7fbd68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:14.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.220+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3740861703 shutdown_connections 2024-03-25T19:30:14.221 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.220+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3740861703 wait complete. 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.221+0000 7fbd6dee2640 10 monclient: build_initial_monmap 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.221+0000 7fbd6dee2640 10 monclient: monmap: 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.221+0000 7fbd6dee2640 1 librados: starting msgr at 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.221+0000 7fbd6dee2640 1 librados: starting objecter 2024-03-25T19:30:14.222 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 1 Processor -- start 2024-03-25T19:30:14.223 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 1 -- start start 2024-03-25T19:30:14.223 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 1 librados: setting wanted keys 2024-03-25T19:30:14.223 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 1 librados: calling monclient init 2024-03-25T19:30:14.223 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 10 monclient: init 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 10 monclient(hunting): picked mon.noname-a con 0x7fbd680b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 10 monclient(hunting): picked mon.noname-c con 0x7fbd680b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 10 monclient(hunting): picked mon.noname-b con 0x7fbd6814be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd681dad80 con 0x7fbd680b1730 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd68143900 con 0x7fbd6814be90 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd68143a90 con 0x7fbd680b4d60 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.222+0000 7fbd6dee2640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:14.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.223+0000 7fbd6dee2640 10 monclient(hunting): authenticate will time out at 6941.320880s 2024-03-25T19:30:14.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.232+0000 7fbd67fff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7fbd6814be90 0x7fbd68141550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:51366/0 (socket says 172.21.15.60:51366) 2024-03-25T19:30:14.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.232+0000 7fbd67fff640 1 -- 172.21.15.60:0/3873848300 learned_addr learned my addr 172.21.15.60:0/3873848300 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:14.234 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.234+0000 7fbd4ffff640 10 client.?.objecter ms_handle_connect 0x7fbd680b4d60 2024-03-25T19:30:14.234 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.234+0000 7fbd4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:14.235 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.236+0000 7fbd4ffff640 10 client.?.objecter ms_handle_connect 0x7fbd6814be90 2024-03-25T19:30:14.235 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.236+0000 7fbd4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:14.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.246+0000 7fbd4ffff640 10 client.?.objecter ms_handle_connect 0x7fbd680b1730 2024-03-25T19:30:14.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.247+0000 7fbd4ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:14.249 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.249+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487849594 0 0) 0x7fbd68143900 con 0x7fbd6814be90 2024-03-25T19:30:14.249 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.250+0000 7fbd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:14.249 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.250+0000 7fbd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:14.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.250+0000 7fbd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:14.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.250+0000 7fbd4ffff640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:30:14.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.250+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd54004eb0 con 0x7fbd6814be90 2024-03-25T19:30:14.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.254+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118998700 0 0) 0x7fbd68143a90 con 0x7fbd680b4d60 2024-03-25T19:30:14.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.254+0000 7fbd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:14.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.254+0000 7fbd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:14.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.254+0000 7fbd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:14.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.254+0000 7fbd4ffff640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:30:14.253 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.254+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd68143900 con 0x7fbd680b4d60 2024-03-25T19:30:14.259 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2469617285 0 0) 0x7fbd68143900 con 0x7fbd680b4d60 2024-03-25T19:30:14.259 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd68143a90 con 0x7fbd680b4d60 2024-03-25T19:30:14.259 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd5c001b00 con 0x7fbd680b4d60 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750487033 0 0) 0x7fbd681dad80 con 0x7fbd680b1730 2024-03-25T19:30:14.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd68143900 con 0x7fbd680b1730 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842428248 0 0) 0x7fbd54004eb0 con 0x7fbd6814be90 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd681dad80 con 0x7fbd6814be90 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd60001c70 con 0x7fbd6814be90 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.259+0000 7fbd4ffff640 10 monclient(hunting): dump: 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:14.261 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.262 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.262 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:14.262 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:14.262 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:14.262 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:14.262 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:14.262 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 134980946 0 0) 0x7fbd681dad80 con 0x7fbd6814be90 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 >> v1:172.21.15.60:6790/0 conn(0x7fbd680b4d60 legacy=0x7fbd6812fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 >> v1:172.21.15.60:6789/0 conn(0x7fbd680b1730 legacy=0x7fbd681db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 1 monclient: found mon.b 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd681db9b0 con 0x7fbd6814be90 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:14.274 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 10 monclient: _renew_subs 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd68140e90 con 0x7fbd6814be90 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 10 monclient: _renew_subs 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd681307c0 con 0x7fbd6814be90 2024-03-25T19:30:14.275 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.274+0000 7fbd6dee2640 1 librados: init done 2024-03-25T19:30:14.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.275+0000 7fbd6dee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:14.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.275+0000 7fbd6dee2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:14.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.275+0000 7fbd6dee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:14.276 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.275+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd681401f0 con 0x7fbd6814be90 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.553+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd60003990 con 0x7fbd6814be90 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.553+0000 7fbd4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.553+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd60003d20 con 0x7fbd6814be90 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.553+0000 7fbd4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.553+0000 7fbd4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.553+0000 7fbd65ffb640 4 set_mon_vals no callback set 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.553+0000 7fbd4ffff640 10 monclient: dump: 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:14.553 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:14.554 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:14.554 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:14.554 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:14.554 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:14.554 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:14.554 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.554+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fbd60003f40 con 0x7fbd6814be90 2024-03-25T19:30:14.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.554+0000 7fbd677fe640 10 monclient: get_auth_request con 0x7fbd5406df70 auth_method 0 2024-03-25T19:30:14.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.555+0000 7fbd4ffff640 10 client.4663.objecter ms_handle_connect 0x7fbd5406df70 2024-03-25T19:30:14.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 781133481 0 0) 0x7fbd600549e0 con 0x7fbd6814be90 2024-03-25T19:30:14.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 10 client.4663.objecter ms_dispatch 0x7fbd68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:30:14.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:30:14.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:30:14.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T19:30:14.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd600504f0 con 0x7fbd6814be90 2024-03-25T19:30:14.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:14.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.558+0000 7fbd4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:14.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.782+0000 7fbd6dee2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:14.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.782+0000 7fbd6dee2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:14.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.782+0000 7fbd6dee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:14.782 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.782+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd68142da0 con 0x7fbd6814be90 2024-03-25T19:30:14.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.806+0000 7fbd4ffff640 1 -- 172.21.15.60:0/3873848300 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3544483432) 0x7fbd60052420 con 0x7fbd6814be90 2024-03-25T19:30:14.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.806+0000 7fbd4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:14.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.806+0000 7fbd4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:14.806 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:30:14.806 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:20:21.916978+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:20:21.916978+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 10 librados: watch_flush enter 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 10 librados: watch_flush exit 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 10 client.4663.objecter successfully canceled tick 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 >> v1:172.21.15.90:6800/1827638754 conn(0x7fbd5406df70 legacy=0x7fbd54070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 10 monclient: shutdown 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 >> v1:172.21.15.90:6789/0 conn(0x7fbd6814be90 legacy=0x7fbd68141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 shutdown_connections 2024-03-25T19:30:14.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 >> 172.21.15.60:0/3873848300 conn(0x7fbd680617a0 msgr2=0x7fbd680b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:14.810 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 shutdown_connections 2024-03-25T19:30:14.810 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 1 -- 172.21.15.60:0/3873848300 wait complete. 2024-03-25T19:30:14.810 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:14.809+0000 7fbd6dee2640 1 librados: shutdown 2024-03-25T19:30:14.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:21.477 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:22.188 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.479 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:22.191 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.515 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:22.226 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.522 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:22.176 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.527 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:22.239 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.531 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:22.185 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.537 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:22.249 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.558 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:22.212 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.562 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:22.273 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:21.626 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.725+0000 7fbd5f8b3640 10 monclient: get_monmap_and_config 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.725+0000 7fbd5f8b3640 10 monclient: build_initial_monmap 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.725+0000 7fbd5f8b3640 10 monclient: monmap: 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:30:21.726500+0000 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:30:21.726500+0000 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:21.725 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:21.726 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:30:21.726 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:30:21.726 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.726+0000 7fbd5f8b3640 1 Processor -- start 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.726+0000 7fbd5f8b3640 1 -- start start 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.726+0000 7fbd5f8b3640 10 monclient: init 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd580b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 10 monclient(hunting): picked mon.noname-c con 0x7fbd580b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd5814be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:21.727 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580af940 con 0x7fbd5814be90 2024-03-25T19:30:21.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580b0a70 con 0x7fbd580b1730 2024-03-25T19:30:21.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd580b0d50 con 0x7fbd580b4d60 2024-03-25T19:30:21.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:21.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.727+0000 7fbd5f8b3640 10 monclient(hunting): authenticate will time out at 6948.825148s 2024-03-25T19:30:21.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.758+0000 7fbd5d64f640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fbd580b4d60 0x7fbd5813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:42998/0 (socket says 172.21.15.60:42998) 2024-03-25T19:30:21.758 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.758+0000 7fbd5d64f640 1 -- 172.21.15.60:0/4188501872 learned_addr learned my addr 172.21.15.60:0/4188501872 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:21.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.772+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293000544 0 0) 0x7fbd580b0d50 con 0x7fbd580b4d60 2024-03-25T19:30:21.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.772+0000 7fbd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:21.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.772+0000 7fbd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:21.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.772+0000 7fbd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:21.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.772+0000 7fbd4ffff640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:30:21.772 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.772+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd44003520 con 0x7fbd580b4d60 2024-03-25T19:30:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.781+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494951586 0 0) 0x7fbd580b0a70 con 0x7fbd580b1730 2024-03-25T19:30:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.781+0000 7fbd4ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:30:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.781+0000 7fbd4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:21.780 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.781+0000 7fbd4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:21.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.781+0000 7fbd4ffff640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:30:21.781 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.781+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd580b0d50 con 0x7fbd580b1730 2024-03-25T19:30:21.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3902360010 0 0) 0x7fbd44003520 con 0x7fbd580b4d60 2024-03-25T19:30:21.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 >> v1:172.21.15.90:6789/0 conn(0x7fbd580b1730 legacy=0x7fbd580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:21.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 >> v1:172.21.15.60:6789/0 conn(0x7fbd5814be90 legacy=0x7fbd5814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:21.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:30:21.797 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 1 monclient: found mon.noname-b 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd580b1030 con 0x7fbd580b4d60 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 10 monclient: _finish_auth 0 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd54001690 con 0x7fbd580b4d60 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd4ffff640 10 monclient: dump: 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.798 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd5f8b3640 5 monclient: authenticate success, global_id 4661 2024-03-25T19:30:21.799 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.797+0000 7fbd5f8b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd540017f0 con 0x7fbd580b4d60 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd4ffff640 1 -- 172.21.15.60:0/4188501872 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd54001c00 con 0x7fbd580b4d60 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd4ffff640 10 monclient: dump: 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:21.804 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:21.805 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:21.805 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:21.805 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:21.805 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd5f8b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:21.805 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.804+0000 7fbd5f8b3640 4 set_mon_vals no callback set 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.805+0000 7fbd5f8b3640 10 monclient: shutdown 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.805+0000 7fbd5f8b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.805+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/4188501872 >> v1:172.21.15.60:6790/0 conn(0x7fbd580b4d60 legacy=0x7fbd5813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.805+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/4188501872 shutdown_connections 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.805+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/4188501872 >> 172.21.15.60:0/4188501872 conn(0x7fbd580617a0 msgr2=0x7fbd58061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.805+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/4188501872 shutdown_connections 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.805+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/4188501872 wait complete. 2024-03-25T19:30:21.806 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 10 monclient: build_initial_monmap 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 10 monclient: monmap: 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 1 librados: starting msgr at 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 1 librados: starting objecter 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 1 Processor -- start 2024-03-25T19:30:21.807 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 1 -- start start 2024-03-25T19:30:21.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 1 librados: setting wanted keys 2024-03-25T19:30:21.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 1 librados: calling monclient init 2024-03-25T19:30:21.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.806+0000 7fbd5f8b3640 10 monclient: init 2024-03-25T19:30:21.808 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd580b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 10 monclient(hunting): picked mon.noname-c con 0x7fbd580b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd5814be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd58138010 con 0x7fbd580b1730 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd581382f0 con 0x7fbd5814be90 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd58136d90 con 0x7fbd580b4d60 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:21.809 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.807+0000 7fbd5f8b3640 10 monclient(hunting): authenticate will time out at 6948.905335s 2024-03-25T19:30:21.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.816+0000 7fbd5ce4e640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7fbd580b4d60 0x7fbd58143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:43018/0 (socket says 172.21.15.60:43018) 2024-03-25T19:30:21.816 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.817+0000 7fbd5ce4e640 1 -- 172.21.15.60:0/2615254057 learned_addr learned my addr 172.21.15.60:0/2615254057 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:21.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.819+0000 7fbd4dffb640 10 client.?.objecter ms_handle_connect 0x7fbd5814be90 2024-03-25T19:30:21.819 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.819+0000 7fbd4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:21.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.821+0000 7fbd4dffb640 10 client.?.objecter ms_handle_connect 0x7fbd580b4d60 2024-03-25T19:30:21.820 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.821+0000 7fbd4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:21.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.832+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3287651812 0 0) 0x7fbd58136d90 con 0x7fbd580b4d60 2024-03-25T19:30:21.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.832+0000 7fbd4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:21.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.832+0000 7fbd4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:21.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.832+0000 7fbd4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:21.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.832+0000 7fbd4dffb640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:30:21.832 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.832+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c004eb0 con 0x7fbd580b4d60 2024-03-25T19:30:21.833 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.834+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832149730 0 0) 0x7fbd581382f0 con 0x7fbd5814be90 2024-03-25T19:30:21.833 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.834+0000 7fbd4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:21.833 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.834+0000 7fbd4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:21.833 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.834+0000 7fbd4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:21.833 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.834+0000 7fbd4dffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:30:21.833 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.834+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd58136d90 con 0x7fbd5814be90 2024-03-25T19:30:21.834 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.835+0000 7fbd4dffb640 10 client.?.objecter ms_handle_connect 0x7fbd580b1730 2024-03-25T19:30:21.834 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.835+0000 7fbd4dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:21.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.851+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085472210 0 0) 0x7fbd58138010 con 0x7fbd580b1730 2024-03-25T19:30:21.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.851+0000 7fbd4dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:30:21.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.851+0000 7fbd4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:21.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.851+0000 7fbd4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:21.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.851+0000 7fbd4dffb640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:30:21.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.851+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd581382f0 con 0x7fbd580b1730 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.858+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990268880 0 0) 0x7fbd581382f0 con 0x7fbd580b1730 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.858+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd58138010 con 0x7fbd580b1730 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.858+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd54001b00 con 0x7fbd580b1730 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.858+0000 7fbd4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.858+0000 7fbd4dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.858+0000 7fbd4dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:21.858 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:21.859 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.858+0000 7fbd4dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:21.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2552182492 0 0) 0x7fbd3c004eb0 con 0x7fbd580b4d60 2024-03-25T19:30:21.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd581382f0 con 0x7fbd580b4d60 2024-03-25T19:30:21.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd48001a10 con 0x7fbd580b4d60 2024-03-25T19:30:21.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:21.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:21.869 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 895467263 0 0) 0x7fbd58136d90 con 0x7fbd5814be90 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd3c004eb0 con 0x7fbd5814be90 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd50001b00 con 0x7fbd5814be90 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.869+0000 7fbd4dffb640 10 monclient(hunting): dump: 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.870 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:21.871 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:21.871 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:21.871 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:21.871 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:21.871 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:21.871 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:21.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.877+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 201119785 0 0) 0x7fbd58138010 con 0x7fbd580b1730 2024-03-25T19:30:21.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.877+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 >> v1:172.21.15.60:6790/0 conn(0x7fbd580b4d60 legacy=0x7fbd58143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:21.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.877+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 >> v1:172.21.15.90:6789/0 conn(0x7fbd5814be90 legacy=0x7fbd581427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:21.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.877+0000 7fbd4dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:30:21.877 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.877+0000 7fbd4dffb640 1 monclient: found mon.a 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.877+0000 7fbd4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd4dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58144010 con 0x7fbd580b1730 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd4dffb640 10 monclient: _finish_auth 0 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd4dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 5 monclient: authenticate success, global_id 4683 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 10 monclient: _renew_subs 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd581372b0 con 0x7fbd580b1730 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 10 monclient: _renew_subs 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:21.878 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd581369c0 con 0x7fbd580b1730 2024-03-25T19:30:21.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.878+0000 7fbd5f8b3640 1 librados: init done 2024-03-25T19:30:21.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.879+0000 7fbd5f8b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:21.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.879+0000 7fbd5f8b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:21.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.879+0000 7fbd5f8b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:21.879 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:21.879+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd58136f20 con 0x7fbd580b1730 2024-03-25T19:30:22.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd54001210 con 0x7fbd580b1730 2024-03-25T19:30:22.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:22.203 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7fbd54004c30 con 0x7fbd580b1730 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4dffb640 10 monclient: dump: 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:22.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4f7fe640 4 set_mon_vals no callback set 2024-03-25T19:30:22.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.204+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7fbd5401ac00 con 0x7fbd580b1730 2024-03-25T19:30:22.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.205+0000 7fbd5ce4e640 10 monclient: get_auth_request con 0x7fbd3c06e0b0 auth_method 0 2024-03-25T19:30:22.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.205+0000 7fbd4dffb640 10 client.4683.objecter ms_handle_connect 0x7fbd3c06e0b0 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7fbd54054840 con 0x7fbd580b1730 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 10 client.4683.objecter ms_dispatch 0x7fbd58000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd54050180 con 0x7fbd580b1730 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:22.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.213+0000 7fbd4dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:22.485 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.485+0000 7fbd5f8b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:22.485 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.485+0000 7fbd5f8b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:22.485 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.485+0000 7fbd5f8b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:22.485 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.485+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd580627b0 con 0x7fbd580b1730 2024-03-25T19:30:22.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.526+0000 7fbd4dffb640 1 -- 172.21.15.60:0/2615254057 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7fbd54052280 con 0x7fbd580b1730 2024-03-25T19:30:22.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.526+0000 7fbd4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:22.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.526+0000 7fbd4dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:22.526 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:30:22.526 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:22.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 10 librados: watch_flush enter 2024-03-25T19:30:22.528 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 10 librados: watch_flush exit 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 10 client.4683.objecter successfully canceled tick 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 >> v1:172.21.15.90:6800/1827638754 conn(0x7fbd3c06e0b0 legacy=0x7fbd3c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 10 monclient: shutdown 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 >> v1:172.21.15.60:6789/0 conn(0x7fbd580b1730 legacy=0x7fbd58135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.528+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 shutdown_connections 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.529+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 >> 172.21.15.60:0/2615254057 conn(0x7fbd580617a0 msgr2=0x7fbd580b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.529+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 shutdown_connections 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.529+0000 7fbd5f8b3640 1 -- 172.21.15.60:0/2615254057 wait complete. 2024-03-25T19:30:22.529 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:22.529+0000 7fbd5f8b3640 1 librados: shutdown 2024-03-25T19:30:22.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:29.347 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:29.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.466+0000 7f9f42d3a640 10 monclient: get_monmap_and_config 2024-03-25T19:30:29.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.466+0000 7f9f42d3a640 10 monclient: build_initial_monmap 2024-03-25T19:30:29.466 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.467+0000 7f9f42d3a640 10 monclient: monmap: 2024-03-25T19:30:29.466 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:30:29.467703+0000 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:30:29.467703+0000 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:30:29.467 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:29.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.467+0000 7f9f42d3a640 1 Processor -- start 2024-03-25T19:30:29.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.468+0000 7f9f42d3a640 1 -- start start 2024-03-25T19:30:29.468 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.468+0000 7f9f42d3a640 10 monclient: init 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.468+0000 7f9f42d3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 10 monclient(hunting): picked mon.noname-c con 0x7f9f3c06c380 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3c06d670 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3c0af8f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c06c9a0 con 0x7f9f3c0af8f0 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0ada90 con 0x7f9f3c06c380 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c0b0d10 con 0x7f9f3c06d670 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:29.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.469+0000 7f9f42d3a640 10 monclient(hunting): authenticate will time out at 6956.567114s 2024-03-25T19:30:29.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.476+0000 7f9f3bfff640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f9f3c06d670 0x7f9f3c0ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:57134/0 (socket says 172.21.15.60:57134) 2024-03-25T19:30:29.475 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.476+0000 7f9f3bfff640 1 -- 172.21.15.60:0/3233440633 learned_addr learned my addr 172.21.15.60:0/3233440633 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:29.487 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.487+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232813857 0 0) 0x7f9f3c0ada90 con 0x7f9f3c06c380 2024-03-25T19:30:29.487 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.487+0000 7f9f3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:29.487 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.487+0000 7f9f3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:29.487 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.487+0000 7f9f3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:29.488 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.487+0000 7f9f3b7fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:30:29.488 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.487+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f2c003520 con 0x7f9f3c06c380 2024-03-25T19:30:29.488 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.488+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553917589 0 0) 0x7f9f3c0b0d10 con 0x7f9f3c06d670 2024-03-25T19:30:29.488 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.488+0000 7f9f3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:29.488 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.488+0000 7f9f3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:29.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.488+0000 7f9f3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:29.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.488+0000 7f9f3b7fe640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:30:29.489 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.488+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3c0ada90 con 0x7f9f3c06d670 2024-03-25T19:30:29.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 725117761 0 0) 0x7f9f3c0ada90 con 0x7f9f3c06d670 2024-03-25T19:30:29.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 >> v1:172.21.15.90:6789/0 conn(0x7f9f3c06c380 legacy=0x7f9f3c0b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:29.497 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 >> v1:172.21.15.60:6789/0 conn(0x7f9f3c0af8f0 legacy=0x7f9f3c0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 1 monclient: found mon.noname-b 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c0add00 con 0x7f9f3c06d670 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f28001670 con 0x7f9f3c06d670 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f3b7fe640 10 monclient: dump: 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:29.498 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f42d3a640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:30:29.499 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.498+0000 7f9f42d3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f280017d0 con 0x7f9f3c06d670 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f3b7fe640 1 -- 172.21.15.60:0/3233440633 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f28001ba0 con 0x7f9f3c06d670 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f3b7fe640 10 monclient: dump: 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:29.504 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:29.505 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:29.505 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:29.505 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:29.505 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:29.505 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f42d3a640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:29.505 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.504+0000 7f9f42d3a640 4 set_mon_vals no callback set 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.505+0000 7f9f42d3a640 10 monclient: shutdown 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.505+0000 7f9f42d3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.505+0000 7f9f42d3a640 1 -- 172.21.15.60:0/3233440633 >> v1:172.21.15.60:6790/0 conn(0x7f9f3c06d670 legacy=0x7f9f3c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.505+0000 7f9f42d3a640 1 -- 172.21.15.60:0/3233440633 shutdown_connections 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.505+0000 7f9f42d3a640 1 -- 172.21.15.60:0/3233440633 >> 172.21.15.60:0/3233440633 conn(0x7f9f3c0b5c00 msgr2=0x7f9f3c0b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.505+0000 7f9f42d3a640 1 -- 172.21.15.60:0/3233440633 shutdown_connections 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.505+0000 7f9f42d3a640 1 -- 172.21.15.60:0/3233440633 wait complete. 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.506+0000 7f9f42d3a640 10 monclient: build_initial_monmap 2024-03-25T19:30:29.506 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.506+0000 7f9f42d3a640 10 monclient: monmap: 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.507+0000 7f9f42d3a640 1 librados: starting msgr at 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.507+0000 7f9f42d3a640 1 librados: starting objecter 2024-03-25T19:30:29.507 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.507+0000 7f9f42d3a640 1 Processor -- start 2024-03-25T19:30:29.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.507+0000 7f9f42d3a640 1 -- start start 2024-03-25T19:30:29.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.507+0000 7f9f42d3a640 1 librados: setting wanted keys 2024-03-25T19:30:29.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.507+0000 7f9f42d3a640 1 librados: calling monclient init 2024-03-25T19:30:29.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.507+0000 7f9f42d3a640 10 monclient: init 2024-03-25T19:30:29.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:29.508 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3c06c380 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 10 monclient(hunting): picked mon.noname-c con 0x7f9f3c06d670 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3c0af8f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c146430 con 0x7f9f3c0af8f0 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c147470 con 0x7f9f3c06c380 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3c146660 con 0x7f9f3c06d670 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:29.509 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.508+0000 7f9f42d3a640 10 monclient(hunting): authenticate will time out at 6956.606117s 2024-03-25T19:30:29.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.525+0000 7f9f40ad6640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9f3c06c380 0x7f9f3c146d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:58204/0 (socket says 172.21.15.60:58204) 2024-03-25T19:30:29.525 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.525+0000 7f9f40ad6640 1 -- 172.21.15.60:0/2163031285 learned_addr learned my addr 172.21.15.60:0/2163031285 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:29.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.526+0000 7f9f397fa640 10 client.?.objecter ms_handle_connect 0x7f9f3c06c380 2024-03-25T19:30:29.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.526+0000 7f9f397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:29.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.526+0000 7f9f397fa640 10 client.?.objecter ms_handle_connect 0x7f9f3c0af8f0 2024-03-25T19:30:29.526 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.526+0000 7f9f397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:29.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.542+0000 7f9f397fa640 10 client.?.objecter ms_handle_connect 0x7f9f3c06d670 2024-03-25T19:30:29.541 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.542+0000 7f9f397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:29.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.544+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350785056 0 0) 0x7f9f3c147470 con 0x7f9f3c06c380 2024-03-25T19:30:29.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.544+0000 7f9f397fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:29.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.544+0000 7f9f397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:29.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.544+0000 7f9f397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:29.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.544+0000 7f9f397fa640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:30:29.543 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.544+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20004eb0 con 0x7f9f3c06c380 2024-03-25T19:30:29.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.548+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429097055 0 0) 0x7f9f3c146430 con 0x7f9f3c0af8f0 2024-03-25T19:30:29.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.548+0000 7f9f397fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:29.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.548+0000 7f9f397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:29.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.548+0000 7f9f397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:29.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.548+0000 7f9f397fa640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:30:29.548 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.548+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3c147470 con 0x7f9f3c0af8f0 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.556+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 552135448 0 0) 0x7f9f20004eb0 con 0x7f9f3c06c380 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.556+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f3c146430 con 0x7f9f3c06c380 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.556+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f30001990 con 0x7f9f3c06c380 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.556+0000 7f9f397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.556+0000 7f9f397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.556+0000 7f9f397fa640 10 monclient(hunting): dump: 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:29.556 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:29.557 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:29.557 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:29.557 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:29.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.556+0000 7f9f397fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:29.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.559+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736114684 0 0) 0x7f9f3c146660 con 0x7f9f3c06d670 2024-03-25T19:30:29.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.559+0000 7f9f397fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:29.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.559+0000 7f9f397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:29.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.559+0000 7f9f397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:29.558 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.559+0000 7f9f397fa640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:30:29.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.559+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20004eb0 con 0x7f9f3c06d670 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 281036072 0 0) 0x7f9f3c146430 con 0x7f9f3c06c380 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 >> v1:172.21.15.60:6790/0 conn(0x7f9f3c06d670 legacy=0x7f9f3c1dee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 >> v1:172.21.15.60:6789/0 conn(0x7f9f3c0af8f0 legacy=0x7f9f3c1e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 1 monclient: found mon.b 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:29.579 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3c146940 con 0x7f9f3c06c380 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 10 monclient: _renew_subs 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f3c148690 con 0x7f9f3c06c380 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 10 monclient: _renew_subs 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f3c0713e0 con 0x7f9f3c06c380 2024-03-25T19:30:29.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.579+0000 7f9f42d3a640 1 librados: init done 2024-03-25T19:30:29.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.580+0000 7f9f42d3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:29.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.580+0000 7f9f42d3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:29.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.580+0000 7f9f42d3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:29.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.580+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3c0b13b0 con 0x7f9f3c06c380 2024-03-25T19:30:29.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.866+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f30003610 con 0x7f9f3c06c380 2024-03-25T19:30:29.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.866+0000 7f9f397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:29.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.866+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f300039e0 con 0x7f9f3c06c380 2024-03-25T19:30:29.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.866+0000 7f9f397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:29.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.866+0000 7f9f397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:29.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.866+0000 7f9f397fa640 10 monclient: dump: 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:29.867 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.866+0000 7f9f3affd640 4 set_mon_vals no callback set 2024-03-25T19:30:29.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.867+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f9f30003bc0 con 0x7f9f3c06c380 2024-03-25T19:30:29.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.868+0000 7f9f3bfff640 10 monclient: get_auth_request con 0x7f9f2006de00 auth_method 0 2024-03-25T19:30:29.868 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.868+0000 7f9f397fa640 10 client.4675.objecter ms_handle_connect 0x7f9f2006de00 2024-03-25T19:30:29.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.884+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f9f30054830 con 0x7f9f3c06c380 2024-03-25T19:30:29.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.884+0000 7f9f397fa640 10 client.4675.objecter ms_dispatch 0x7f9f3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:29.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.884+0000 7f9f397fa640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:29.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.884+0000 7f9f397fa640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:29.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.884+0000 7f9f397fa640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T19:30:29.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.886+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f30050340 con 0x7f9f3c06c380 2024-03-25T19:30:29.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.886+0000 7f9f397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:29.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:29.886+0000 7f9f397fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:30.116 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.116+0000 7f9f42d3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:30.116 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.116+0000 7f9f42d3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:30.116 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.116+0000 7f9f42d3a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:30.116 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.116+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f3c144390 con 0x7f9f3c06c380 2024-03-25T19:30:30.148 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.149+0000 7f9f397fa640 1 -- 172.21.15.60:0/2163031285 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f9f30052270 con 0x7f9f3c06c380 2024-03-25T19:30:30.149 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.149+0000 7f9f397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:30.149 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.149+0000 7f9f397fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:30.149 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:30:30.149 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.151+0000 7f9f42d3a640 10 librados: watch_flush enter 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.151+0000 7f9f42d3a640 10 librados: watch_flush exit 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.151+0000 7f9f42d3a640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.151+0000 7f9f42d3a640 10 client.4675.objecter successfully canceled tick 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 >> v1:172.21.15.90:6800/1827638754 conn(0x7f9f2006de00 legacy=0x7f9f200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 10 monclient: shutdown 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:30.151 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 >> v1:172.21.15.90:6789/0 conn(0x7f9f3c06c380 legacy=0x7f9f3c146d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:30.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 shutdown_connections 2024-03-25T19:30:30.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 >> 172.21.15.60:0/2163031285 conn(0x7f9f3c0b5c00 msgr2=0x7f9f3c0ae680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:30.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 shutdown_connections 2024-03-25T19:30:30.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 1 -- 172.21.15.60:0/2163031285 wait complete. 2024-03-25T19:30:30.152 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:30.152+0000 7f9f42d3a640 1 librados: shutdown 2024-03-25T19:30:30.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:36.969 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:37.085 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.086+0000 7f348323e640 10 monclient: get_monmap_and_config 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.086+0000 7f348323e640 10 monclient: build_initial_monmap 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.086+0000 7f348323e640 10 monclient: monmap: 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:30:37.086848+0000 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:30:37.086848+0000 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:30:37.086 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.086+0000 7f348323e640 1 Processor -- start 2024-03-25T19:30:37.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.086+0000 7f348323e640 1 -- start start 2024-03-25T19:30:37.087 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.086+0000 7f348323e640 10 monclient: init 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 10 monclient(hunting): picked mon.noname-c con 0x7f347c0b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 10 monclient(hunting): picked mon.noname-b con 0x7f347c0b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 10 monclient(hunting): picked mon.noname-a con 0x7f347c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0af940 con 0x7f347c14be90 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0b0a70 con 0x7f347c0b4d60 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c0b0d50 con 0x7f347c0b1730 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:37.088 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.087+0000 7f348323e640 10 monclient(hunting): authenticate will time out at 6964.185662s 2024-03-25T19:30:37.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.097+0000 7f3480fda640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f347c0b4d60 0x7f347c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:54690/0 (socket says 172.21.15.60:54690) 2024-03-25T19:30:37.096 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.097+0000 7f3480fda640 1 -- 172.21.15.60:0/4077465488 learned_addr learned my addr 172.21.15.60:0/4077465488 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:37.111 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.112+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551070453 0 0) 0x7f347c0b0a70 con 0x7f347c0b4d60 2024-03-25T19:30:37.111 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.112+0000 7f347b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:37.111 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.112+0000 7f347b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:37.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.112+0000 7f347b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:37.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.112+0000 7f347b7fe640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:30:37.112 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.112+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f346c003520 con 0x7f347c0b4d60 2024-03-25T19:30:37.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.123+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734887055 0 0) 0x7f347c0b0d50 con 0x7f347c0b1730 2024-03-25T19:30:37.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.123+0000 7f347b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:37.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.123+0000 7f347b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:37.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.123+0000 7f347b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:37.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.123+0000 7f347b7fe640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:30:37.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.123+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c0b0a70 con 0x7f347c0b1730 2024-03-25T19:30:37.123 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.124+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633646155 0 0) 0x7f347c0af940 con 0x7f347c14be90 2024-03-25T19:30:37.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.124+0000 7f347b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:30:37.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.124+0000 7f347b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:37.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.124+0000 7f347b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:37.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.124+0000 7f347b7fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:30:37.124 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.124+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c0b0d50 con 0x7f347c14be90 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705909913 0 0) 0x7f346c003520 con 0x7f347c0b4d60 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 >> v1:172.21.15.60:6790/0 conn(0x7f347c0b1730 legacy=0x7f347c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 >> v1:172.21.15.60:6789/0 conn(0x7f347c14be90 legacy=0x7f347c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 1 monclient: found mon.noname-c 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c0b1030 con 0x7f347c0b4d60 2024-03-25T19:30:37.128 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3470001690 con 0x7f347c0b4d60 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f348323e640 5 monclient: authenticate success, global_id 4678 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f348323e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f347b7fe640 10 monclient: dump: 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:37.129 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:37.130 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:37.130 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:37.130 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.130 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.128+0000 7f348323e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34700017f0 con 0x7f347c0b4d60 2024-03-25T19:30:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f347b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:37.135 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f347b7fe640 1 -- 172.21.15.60:0/4077465488 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3470001c00 con 0x7f347c0b4d60 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f347b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f347b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f347b7fe640 10 monclient: dump: 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f348323e640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:37.136 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.136+0000 7f348323e640 4 set_mon_vals no callback set 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.137+0000 7f348323e640 10 monclient: shutdown 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.137+0000 7f348323e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.137+0000 7f348323e640 1 -- 172.21.15.60:0/4077465488 >> v1:172.21.15.90:6789/0 conn(0x7f347c0b4d60 legacy=0x7f347c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.137+0000 7f348323e640 1 -- 172.21.15.60:0/4077465488 shutdown_connections 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.137+0000 7f348323e640 1 -- 172.21.15.60:0/4077465488 >> 172.21.15.60:0/4077465488 conn(0x7f347c0617a0 msgr2=0x7f347c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.137+0000 7f348323e640 1 -- 172.21.15.60:0/4077465488 shutdown_connections 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.137+0000 7f348323e640 1 -- 172.21.15.60:0/4077465488 wait complete. 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 10 monclient: build_initial_monmap 2024-03-25T19:30:37.138 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 10 monclient: monmap: 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 1 librados: starting msgr at 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 1 librados: starting objecter 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 1 Processor -- start 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 1 -- start start 2024-03-25T19:30:37.139 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 1 librados: setting wanted keys 2024-03-25T19:30:37.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 1 librados: calling monclient init 2024-03-25T19:30:37.140 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.138+0000 7f348323e640 10 monclient: init 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 10 monclient(hunting): picked mon.noname-c con 0x7f347c0b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 10 monclient(hunting): picked mon.noname-a con 0x7f347c0b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 10 monclient(hunting): picked mon.noname-b con 0x7f347c14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c1dba90 con 0x7f347c0b4d60 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c1dbc20 con 0x7f347c14be90 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f347c13b560 con 0x7f347c0b1730 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:37.141 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.139+0000 7f348323e640 10 monclient(hunting): authenticate will time out at 6964.237690s 2024-03-25T19:30:37.147 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.147+0000 7f34817db640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f347c14be90 0x7f347c1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:54734/0 (socket says 172.21.15.60:54734) 2024-03-25T19:30:37.147 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.147+0000 7f34817db640 1 -- 172.21.15.60:0/3767815496 learned_addr learned my addr 172.21.15.60:0/3767815496 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:37.147 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.148+0000 7f34797fa640 10 client.?.objecter ms_handle_connect 0x7f347c14be90 2024-03-25T19:30:37.147 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.148+0000 7f34797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:37.148 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.148+0000 7f34797fa640 10 client.?.objecter ms_handle_connect 0x7f347c0b1730 2024-03-25T19:30:37.148 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.148+0000 7f34797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:37.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.156+0000 7f34797fa640 10 client.?.objecter ms_handle_connect 0x7f347c0b4d60 2024-03-25T19:30:37.156 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.156+0000 7f34797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:37.157 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.157+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474238174 0 0) 0x7f347c1dbc20 con 0x7f347c14be90 2024-03-25T19:30:37.157 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.157+0000 7f34797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:37.157 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.157+0000 7f34797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:37.157 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.157+0000 7f34797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:37.157 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.157+0000 7f34797fa640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:30:37.157 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.157+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3460004eb0 con 0x7f347c14be90 2024-03-25T19:30:37.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.163+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306632165 0 0) 0x7f347c1dba90 con 0x7f347c0b4d60 2024-03-25T19:30:37.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.163+0000 7f34797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:37.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.163+0000 7f34797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:37.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.163+0000 7f34797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:37.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.163+0000 7f34797fa640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:30:37.163 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.163+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c1dbc20 con 0x7f347c0b4d60 2024-03-25T19:30:37.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.169+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416625796 0 0) 0x7f347c13b560 con 0x7f347c0b1730 2024-03-25T19:30:37.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.169+0000 7f34797fa640 10 monclient(hunting): _negotiate 2024-03-25T19:30:37.168 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.169+0000 7f34797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:37.169 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.169+0000 7f34797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:37.169 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.169+0000 7f34797fa640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:30:37.169 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.169+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f347c1dba90 con 0x7f347c0b1730 2024-03-25T19:30:37.174 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.175+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1636745705 0 0) 0x7f3460004eb0 con 0x7f347c14be90 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.175+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f347c13b560 con 0x7f347c14be90 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.175+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3474001c70 con 0x7f347c14be90 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.175+0000 7f34797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.175+0000 7f34797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.175+0000 7f34797fa640 10 monclient(hunting): dump: 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:37.175 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:37.176 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.176 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.175+0000 7f34797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.180+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2899901823 0 0) 0x7f347c1dbc20 con 0x7f347c0b4d60 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.180+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3460004eb0 con 0x7f347c0b4d60 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.180+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3468001970 con 0x7f347c0b4d60 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.180+0000 7f34797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.180+0000 7f34797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.180+0000 7f34797fa640 10 monclient(hunting): dump: 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:37.180 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:37.181 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:37.181 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.183+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 137886531 0 0) 0x7f347c1dba90 con 0x7f347c0b1730 2024-03-25T19:30:37.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.184+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f347c1dbc20 con 0x7f347c0b1730 2024-03-25T19:30:37.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.184+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3470001b00 con 0x7f347c0b1730 2024-03-25T19:30:37.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.184+0000 7f34797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.184+0000 7f34797fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.184+0000 7f34797fa640 10 monclient(hunting): dump: 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:37.184 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1681259118 0 0) 0x7f347c13b560 con 0x7f347c14be90 2024-03-25T19:30:37.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 >> v1:172.21.15.60:6790/0 conn(0x7f347c0b1730 legacy=0x7f347c1db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:37.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 >> v1:172.21.15.60:6789/0 conn(0x7f347c0b4d60 legacy=0x7f347c142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:37.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 10 monclient: _finish_hunting 0 2024-03-25T19:30:37.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 1 monclient: found mon.b 2024-03-25T19:30:37.191 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347c13b6f0 con 0x7f347c14be90 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 10 monclient: _finish_auth 0 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 10 monclient: _check_auth_tickets 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f34797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 10 monclient: _renew_subs 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f347c143430 con 0x7f347c14be90 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 10 monclient: _renew_subs 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f347c142610 con 0x7f347c14be90 2024-03-25T19:30:37.192 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.191+0000 7f348323e640 1 librados: init done 2024-03-25T19:30:37.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.193+0000 7f348323e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:37.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.193+0000 7f348323e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:37.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.193+0000 7f348323e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:37.193 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.193+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347c143730 con 0x7f347c14be90 2024-03-25T19:30:37.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.463+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3474003990 con 0x7f347c14be90 2024-03-25T19:30:37.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.463+0000 7f34797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:37.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.463+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3474003c00 con 0x7f347c14be90 2024-03-25T19:30:37.463 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.463+0000 7f34797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.463+0000 7f347affd640 4 set_mon_vals no callback set 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.463+0000 7f34797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.463+0000 7f34797fa640 10 monclient: dump: 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:37.464 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:37.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.464+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f3474003e20 con 0x7f347c14be90 2024-03-25T19:30:37.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.465+0000 7f3480fda640 10 monclient: get_auth_request con 0x7f346006e110 auth_method 0 2024-03-25T19:30:37.465 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.465+0000 7f34797fa640 10 client.4681.objecter ms_handle_connect 0x7f346006e110 2024-03-25T19:30:37.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.469+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f3474054870 con 0x7f347c14be90 2024-03-25T19:30:37.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.469+0000 7f34797fa640 10 client.4681.objecter ms_dispatch 0x7f347c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:37.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.469+0000 7f34797fa640 3 client.4681.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:37.469 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.469+0000 7f34797fa640 3 client.4681.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:37.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.470+0000 7f34797fa640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T19:30:37.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.470+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3474050380 con 0x7f347c14be90 2024-03-25T19:30:37.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.470+0000 7f34797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:37.470 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.470+0000 7f34797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:37.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.724+0000 7f348323e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:37.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.724+0000 7f348323e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:37.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.724+0000 7f348323e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:37.724 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.724+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f347c0627b0 con 0x7f347c14be90 2024-03-25T19:30:37.891 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.891+0000 7f34797fa640 1 -- 172.21.15.60:0/3767815496 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f34740522b0 con 0x7f347c14be90 2024-03-25T19:30:37.891 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.892+0000 7f34797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:37.891 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.892+0000 7f34797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:37.891 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:30:37.892 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 10 librados: watch_flush enter 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 10 librados: watch_flush exit 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 10 client.4681.objecter successfully canceled tick 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 >> v1:172.21.15.90:6800/1827638754 conn(0x7f346006e110 legacy=0x7f34600705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 10 monclient: shutdown 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:37.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 >> v1:172.21.15.90:6789/0 conn(0x7f347c14be90 legacy=0x7f347c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:37.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 shutdown_connections 2024-03-25T19:30:37.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.894+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 >> 172.21.15.60:0/3767815496 conn(0x7f347c0617a0 msgr2=0x7f347c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:37.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.895+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 shutdown_connections 2024-03-25T19:30:37.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.895+0000 7f348323e640 1 -- 172.21.15.60:0/3767815496 wait complete. 2024-03-25T19:30:37.895 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:37.895+0000 7f348323e640 1 librados: shutdown 2024-03-25T19:30:37.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:41.382 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:42.093 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.393 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:42.104 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.433 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:10:42.145 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.453 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:42.107 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.462 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:42.116 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.468 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:42.179 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.473 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:42.184 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.483 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:10:42.137 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:41.503 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:10:42.215 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:30:44.713 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:44.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.824+0000 7f922c7cf640 10 monclient: get_monmap_and_config 2024-03-25T19:30:44.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.825+0000 7f922c7cf640 10 monclient: build_initial_monmap 2024-03-25T19:30:44.824 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.825+0000 7f922c7cf640 10 monclient: monmap: 2024-03-25T19:30:44.824 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:44.824 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:44.824 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:30:44.825711+0000 2024-03-25T19:30:44.825 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:30:44.825711+0000 2024-03-25T19:30:44.825 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:44.825 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:44.825 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:44.825 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:30:44.825 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:30:44.825 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:44.826 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.825+0000 7f922c7cf640 1 Processor -- start 2024-03-25T19:30:44.826 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.825+0000 7f922c7cf640 1 -- start start 2024-03-25T19:30:44.826 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.825+0000 7f922c7cf640 10 monclient: init 2024-03-25T19:30:44.826 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:44.826 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:44.826 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f92240a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:44.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f922413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:44.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f92241430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:44.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9224142520 con 0x7f92241430c0 2024-03-25T19:30:44.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92240b28a0 con 0x7f922413f710 2024-03-25T19:30:44.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92240b2a30 con 0x7f92240a98f0 2024-03-25T19:30:44.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:44.827 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.826+0000 7f922c7cf640 10 monclient(hunting): authenticate will time out at 6971.924665s 2024-03-25T19:30:44.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.839+0000 7f922a56b640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f92240a98f0 0x7f92240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:36408/0 (socket says 172.21.15.60:36408) 2024-03-25T19:30:44.838 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.839+0000 7f922a56b640 1 -- 172.21.15.60:0/1312506456 learned_addr learned my addr 172.21.15.60:0/1312506456 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:44.848 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.849+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392124032 0 0) 0x7f92240b2a30 con 0x7f92240a98f0 2024-03-25T19:30:44.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.849+0000 7f9229569640 10 monclient(hunting): _negotiate 2024-03-25T19:30:44.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.849+0000 7f9229569640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:44.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.849+0000 7f9229569640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:44.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.849+0000 7f9229569640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:30:44.849 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.849+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9210003520 con 0x7f92240a98f0 2024-03-25T19:30:44.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.851+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777086717 0 0) 0x7f9224142520 con 0x7f92241430c0 2024-03-25T19:30:44.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.851+0000 7f9229569640 10 monclient(hunting): _negotiate 2024-03-25T19:30:44.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.851+0000 7f9229569640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:44.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.851+0000 7f9229569640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:44.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.851+0000 7f9229569640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:30:44.850 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.851+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92240b2a30 con 0x7f92241430c0 2024-03-25T19:30:44.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 160369714 0 0) 0x7f9210003520 con 0x7f92240a98f0 2024-03-25T19:30:44.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 >> v1:172.21.15.90:6789/0 conn(0x7f922413f710 legacy=0x7f92241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:44.855 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 >> v1:172.21.15.60:6789/0 conn(0x7f92241430c0 legacy=0x7f922414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 10 monclient: _finish_hunting 0 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 1 monclient: found mon.noname-b 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92240b44c0 con 0x7f92240a98f0 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 10 monclient: _finish_auth 0 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 10 monclient: _check_auth_tickets 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9220001690 con 0x7f92240a98f0 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.855+0000 7f9229569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.856+0000 7f9229569640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.856+0000 7f9229569640 10 monclient: dump: 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:44.856 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.856+0000 7f922c7cf640 5 monclient: authenticate success, global_id 4679 2024-03-25T19:30:44.857 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.856+0000 7f922c7cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92200017f0 con 0x7f92240a98f0 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f9229569640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f9229569640 1 -- 172.21.15.60:0/1312506456 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9220001c00 con 0x7f92240a98f0 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f9229569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f9229569640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f9229569640 10 monclient: dump: 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:44.880 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:44.881 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:44.881 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:44.881 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:44.881 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:44.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f922c7cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:44.881 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.880+0000 7f922c7cf640 4 set_mon_vals no callback set 2024-03-25T19:30:44.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.881+0000 7f922c7cf640 10 monclient: shutdown 2024-03-25T19:30:44.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.881+0000 7f922c7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:44.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.881+0000 7f922c7cf640 1 -- 172.21.15.60:0/1312506456 >> v1:172.21.15.60:6790/0 conn(0x7f92240a98f0 legacy=0x7f92240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:44.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.881+0000 7f922c7cf640 1 -- 172.21.15.60:0/1312506456 shutdown_connections 2024-03-25T19:30:44.882 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.881+0000 7f922c7cf640 1 -- 172.21.15.60:0/1312506456 >> 172.21.15.60:0/1312506456 conn(0x7f92240b7ea0 msgr2=0x7f92240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.881+0000 7f922c7cf640 1 -- 172.21.15.60:0/1312506456 shutdown_connections 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.881+0000 7f922c7cf640 1 -- 172.21.15.60:0/1312506456 wait complete. 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.882+0000 7f922c7cf640 10 monclient: build_initial_monmap 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.882+0000 7f922c7cf640 10 monclient: monmap: 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:44.883 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.882+0000 7f922c7cf640 1 librados: starting msgr at 2024-03-25T19:30:44.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 librados: starting objecter 2024-03-25T19:30:44.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 Processor -- start 2024-03-25T19:30:44.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 -- start start 2024-03-25T19:30:44.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 librados: setting wanted keys 2024-03-25T19:30:44.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 librados: calling monclient init 2024-03-25T19:30:44.884 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient: init 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient(hunting): picked mon.noname-b con 0x7f92240a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient(hunting): picked mon.noname-c con 0x7f922413f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient(hunting): picked mon.noname-a con 0x7f92241430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92241370a0 con 0x7f92241430c0 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9224135090 con 0x7f92240a98f0 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9224135220 con 0x7f922413f710 2024-03-25T19:30:44.885 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:44.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.883+0000 7f922c7cf640 10 monclient(hunting): authenticate will time out at 6971.981834s 2024-03-25T19:30:44.887 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.888+0000 7f922a56b640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f92240a98f0 0x7f92241dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:59318/0 (socket says 172.21.15.60:59318) 2024-03-25T19:30:44.887 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.888+0000 7f922a56b640 1 -- 172.21.15.60:0/27676728 learned_addr learned my addr 172.21.15.60:0/27676728 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:44.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.894+0000 7f921affd640 10 client.?.objecter ms_handle_connect 0x7f92240a98f0 2024-03-25T19:30:44.894 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.894+0000 7f921affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:44.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.900+0000 7f921affd640 10 client.?.objecter ms_handle_connect 0x7f92241430c0 2024-03-25T19:30:44.900 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.900+0000 7f921affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:44.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.904+0000 7f921affd640 10 client.?.objecter ms_handle_connect 0x7f922413f710 2024-03-25T19:30:44.903 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.904+0000 7f921affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:44.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.906+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095842716 0 0) 0x7f9224135090 con 0x7f92240a98f0 2024-03-25T19:30:44.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.906+0000 7f921affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:44.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.906+0000 7f921affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:44.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.906+0000 7f921affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:44.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.906+0000 7f921affd640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:30:44.906 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.906+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9208004eb0 con 0x7f92240a98f0 2024-03-25T19:30:44.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.911+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006677055 0 0) 0x7f92241370a0 con 0x7f92241430c0 2024-03-25T19:30:44.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.911+0000 7f921affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:44.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.911+0000 7f921affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:44.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.911+0000 7f921affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:44.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.911+0000 7f921affd640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:30:44.911 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.911+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9224135090 con 0x7f92241430c0 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.922+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879907010 0 0) 0x7f9208004eb0 con 0x7f92240a98f0 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.922+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92241370a0 con 0x7f92240a98f0 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.922+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9220001c00 con 0x7f92240a98f0 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.922+0000 7f921affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.922+0000 7f921affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.922+0000 7f921affd640 10 monclient(hunting): dump: 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:44.922 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:44.923 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.922+0000 7f921affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:44.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.930+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428475112 0 0) 0x7f9224135220 con 0x7f922413f710 2024-03-25T19:30:44.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.930+0000 7f921affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:44.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.930+0000 7f921affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:44.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.930+0000 7f921affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:44.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.930+0000 7f921affd640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:30:44.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.930+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9208004eb0 con 0x7f922413f710 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 462757018 0 0) 0x7f92241370a0 con 0x7f92240a98f0 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 >> v1:172.21.15.60:6790/0 conn(0x7f922413f710 legacy=0x7f92241d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 >> v1:172.21.15.60:6789/0 conn(0x7f92241430c0 legacy=0x7f92241d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 1 monclient: found mon.b 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92241d5000 con 0x7f92240a98f0 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 10 monclient: _finish_auth 0 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f921affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 5 monclient: authenticate success, global_id 4687 2024-03-25T19:30:44.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 10 monclient: _renew_subs 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92241d52b0 con 0x7f92240a98f0 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 10 monclient: _renew_subs 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9224135910 con 0x7f92240a98f0 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.933+0000 7f922c7cf640 1 librados: init done 2024-03-25T19:30:44.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.934+0000 7f922c7cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:44.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.934+0000 7f922c7cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:44.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.934+0000 7f922c7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:44.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:44.934+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92240627b0 con 0x7f92240a98f0 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.211+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9220004a90 con 0x7f92240a98f0 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.211+0000 7f921affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.211+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9220004e20 con 0x7f92240a98f0 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.211+0000 7f921affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.211+0000 7f921affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.211+0000 7f921affd640 10 monclient: dump: 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:45.211 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:45.212 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:45.212 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:45.212 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:45.212 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.211+0000 7f9228d68640 4 set_mon_vals no callback set 2024-03-25T19:30:45.212 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.212+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f9220005040 con 0x7f92240a98f0 2024-03-25T19:30:45.212 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.213+0000 7f9229d6a640 10 monclient: get_auth_request con 0x7f920806de00 auth_method 0 2024-03-25T19:30:45.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.213+0000 7f921affd640 10 client.4687.objecter ms_handle_connect 0x7f920806de00 2024-03-25T19:30:45.216 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.217+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f92200549b0 con 0x7f92240a98f0 2024-03-25T19:30:45.216 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.217+0000 7f921affd640 10 client.4687.objecter ms_dispatch 0x7f9224000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:45.216 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.217+0000 7f921affd640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:45.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.217+0000 7f921affd640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:45.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.217+0000 7f921affd640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T19:30:45.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.218+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92200504c0 con 0x7f92240a98f0 2024-03-25T19:30:45.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.218+0000 7f921affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:45.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.218+0000 7f921affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:45.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.483+0000 7f922c7cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:45.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.483+0000 7f922c7cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:45.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.483+0000 7f922c7cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:30:45.483 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.483+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f922413bfe0 con 0x7f92240a98f0 2024-03-25T19:30:45.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.514+0000 7f921affd640 1 -- 172.21.15.60:0/27676728 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f92200523f0 con 0x7f92240a98f0 2024-03-25T19:30:45.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.514+0000 7f921affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:45.514 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.514+0000 7f921affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:45.515 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:30:45.515 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 10 librados: watch_flush enter 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 10 librados: watch_flush exit 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 10 client.4687.objecter successfully canceled tick 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 >> v1:172.21.15.90:6800/1827638754 conn(0x7f920806de00 legacy=0x7f92080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 10 monclient: shutdown 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 >> v1:172.21.15.90:6789/0 conn(0x7f92240a98f0 legacy=0x7f92241dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 shutdown_connections 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 >> 172.21.15.60:0/27676728 conn(0x7f92240b7ea0 msgr2=0x7f9224074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:45.517 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 shutdown_connections 2024-03-25T19:30:45.518 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 1 -- 172.21.15.60:0/27676728 wait complete. 2024-03-25T19:30:45.518 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:45.517+0000 7f922c7cf640 1 librados: shutdown 2024-03-25T19:30:45.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:30:52.334 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:30:52.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.447+0000 7f767c4e6640 10 monclient: get_monmap_and_config 2024-03-25T19:30:52.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.447+0000 7f767c4e6640 10 monclient: build_initial_monmap 2024-03-25T19:30:52.446 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.447+0000 7f767c4e6640 10 monclient: monmap: 2024-03-25T19:30:52.446 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:52.446 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:52.446 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:30:52.447743+0000 2024-03-25T19:30:52.446 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:30:52.447743+0000 2024-03-25T19:30:52.447 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:52.447 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:52.447 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:52.447 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:30:52.447 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:30:52.447 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.447+0000 7f767c4e6640 1 Processor -- start 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.447+0000 7f767c4e6640 1 -- start start 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.447+0000 7f767c4e6640 10 monclient: init 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f76740b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f76740b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f767414be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:52.448 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76740af940 con 0x7f767414be90 2024-03-25T19:30:52.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76740b0a70 con 0x7f76740b1730 2024-03-25T19:30:52.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76740b0d50 con 0x7f76740b4d60 2024-03-25T19:30:52.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:52.449 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.448+0000 7f767c4e6640 10 monclient(hunting): authenticate will time out at 6979.546538s 2024-03-25T19:30:52.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.457+0000 7f7679a81640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f76740b1730 0x7f76740b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:34410/0 (socket says 172.21.15.60:34410) 2024-03-25T19:30:52.457 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.457+0000 7f7679a81640 1 -- 172.21.15.60:0/2814863063 learned_addr learned my addr 172.21.15.60:0/2814863063 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:52.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.474+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337821529 0 0) 0x7f76740af940 con 0x7f767414be90 2024-03-25T19:30:52.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.474+0000 7f7679280640 10 monclient(hunting): _negotiate 2024-03-25T19:30:52.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.474+0000 7f7679280640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:52.473 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.474+0000 7f7679280640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:52.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.474+0000 7f7679280640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:30:52.474 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.474+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7660003520 con 0x7f767414be90 2024-03-25T19:30:52.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3938795670 0 0) 0x7f7660003520 con 0x7f767414be90 2024-03-25T19:30:52.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 >> v1:172.21.15.60:6790/0 conn(0x7f76740b4d60 legacy=0x7f767413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:52.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 >> v1:172.21.15.90:6789/0 conn(0x7f76740b1730 legacy=0x7f76740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:52.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 10 monclient: _finish_hunting 0 2024-03-25T19:30:52.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 1 monclient: found mon.noname-a 2024-03-25T19:30:52.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:52.559 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76740b1030 con 0x7f767414be90 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 10 monclient: _finish_auth 0 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 10 monclient: _check_auth_tickets 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f766c001a10 con 0x7f767414be90 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f767c4e6640 5 monclient: authenticate success, global_id 4704 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f767c4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f7679280640 10 monclient: dump: 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.560 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:52.561 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:52.561 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:52.561 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:52.561 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:52.561 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:52.561 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.559+0000 7f767c4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f766c001b70 con 0x7f767414be90 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f7679280640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f7679280640 1 -- 172.21.15.60:0/2814863063 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f766c001f80 con 0x7f767414be90 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f767c4e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f767c4e6640 4 set_mon_vals no callback set 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f7679280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f7679280640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.576+0000 7f7679280640 10 monclient: dump: 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:52.576 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.577 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.577 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:52.577 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:52.577 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:52.577 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:52.577 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:52.577 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:52.578 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.577+0000 7f767c4e6640 10 monclient: shutdown 2024-03-25T19:30:52.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.577+0000 7f767c4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:52.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.577+0000 7f767c4e6640 1 -- 172.21.15.60:0/2814863063 >> v1:172.21.15.60:6789/0 conn(0x7f767414be90 legacy=0x7f767414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:52.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.577+0000 7f767c4e6640 1 -- 172.21.15.60:0/2814863063 shutdown_connections 2024-03-25T19:30:52.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.577+0000 7f767c4e6640 1 -- 172.21.15.60:0/2814863063 >> 172.21.15.60:0/2814863063 conn(0x7f76740617a0 msgr2=0x7f7674061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:52.665 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.577+0000 7f767c4e6640 1 -- 172.21.15.60:0/2814863063 shutdown_connections 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.577+0000 7f767c4e6640 1 -- 172.21.15.60:0/2814863063 wait complete. 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.578+0000 7f767c4e6640 10 monclient: build_initial_monmap 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.578+0000 7f767c4e6640 10 monclient: monmap: 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 librados: starting msgr at 2024-03-25T19:30:52.666 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 librados: starting objecter 2024-03-25T19:30:52.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 Processor -- start 2024-03-25T19:30:52.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 -- start start 2024-03-25T19:30:52.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 librados: setting wanted keys 2024-03-25T19:30:52.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 librados: calling monclient init 2024-03-25T19:30:52.667 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient: init 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f76740b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f76740b4d60 addr v1:172.21.15.60:6789/0 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f767414be90 addr v1:172.21.15.60:6790/0 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f767413bd10 con 0x7f76740b4d60 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76741dadb0 con 0x7f76740b1730 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7674138950 con 0x7f767414be90 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.579+0000 7f767c4e6640 10 monclient(hunting): authenticate will time out at 6979.677794s 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.583+0000 7f7679a81640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f76740b4d60 0x7f767413a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38652/0 (socket says 172.21.15.60:38652) 2024-03-25T19:30:52.670 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.584+0000 7f7679a81640 1 -- 172.21.15.60:0/2355703432 learned_addr learned my addr 172.21.15.60:0/2355703432 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.591+0000 7f766affd640 10 client.?.objecter ms_handle_connect 0x7f76740b4d60 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.591+0000 7f766affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.598+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761863816 0 0) 0x7f767413bd10 con 0x7f76740b4d60 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.598+0000 7f766affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.598+0000 7f766affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.598+0000 7f766affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.598+0000 7f766affd640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.598+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7658004eb0 con 0x7f76740b4d60 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.604+0000 7f766affd640 10 client.?.objecter ms_handle_connect 0x7f76740b1730 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.604+0000 7f766affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.612+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1345610570 0 0) 0x7f7658004eb0 con 0x7f76740b4d60 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.612+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f767413bd10 con 0x7f76740b4d60 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.612+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f76640018b0 con 0x7f76740b4d60 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.612+0000 7f766affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:52.671 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.612+0000 7f766affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.612+0000 7f766affd640 10 monclient(hunting): dump: 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.612+0000 7f766affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.620+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295622109 0 0) 0x7f76741dadb0 con 0x7f76740b1730 2024-03-25T19:30:52.672 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.620+0000 7f766affd640 10 monclient(hunting): _negotiate 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.620+0000 7f766affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.620+0000 7f766affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.620+0000 7f766affd640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.620+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7658004eb0 con 0x7f76740b1730 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.626+0000 7f766affd640 10 client.?.objecter ms_handle_connect 0x7f767414be90 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.626+0000 7f766affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3096030243 0 0) 0x7f767413bd10 con 0x7f76740b4d60 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 >> v1:172.21.15.60:6790/0 conn(0x7f767414be90 legacy=0x7f76741db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 >> v1:172.21.15.90:6789/0 conn(0x7f76740b1730 legacy=0x7f7674138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 10 monclient: _finish_hunting 0 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 1 monclient: found mon.a 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:52.673 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767413ab60 con 0x7f76740b4d60 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 10 monclient: _finish_auth 0 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 10 monclient: _check_auth_tickets 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f766affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 10 monclient: _renew_subs 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.60:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76741db9e0 con 0x7f76740b4d60 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 10 monclient: _renew_subs 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.60:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7674139890 con 0x7f76740b4d60 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.642+0000 7f767c4e6640 1 librados: init done 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.643+0000 7f767c4e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.643+0000 7f767c4e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.643+0000 7f767c4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:52.674 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.643+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f767413acf0 con 0x7f76740b4d60 2024-03-25T19:30:52.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.968+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76640047e0 con 0x7f76740b4d60 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.968+0000 7f766affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.968+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f7664004bf0 con 0x7f76740b4d60 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.968+0000 7f766affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.968+0000 7f766affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.968+0000 7f766affd640 10 monclient: dump: 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:30:52.968 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:30:52.969 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:30:52.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.968+0000 7f7678a7f640 4 set_mon_vals no callback set 2024-03-25T19:30:52.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.969+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f7664004e10 con 0x7f76740b4d60 2024-03-25T19:30:52.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.969+0000 7f767a282640 10 monclient: get_auth_request con 0x7f765806dd90 auth_method 0 2024-03-25T19:30:52.969 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.969+0000 7f766affd640 10 client.4707.objecter ms_handle_connect 0x7f765806dd90 2024-03-25T19:30:52.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f76640547e0 con 0x7f76740b4d60 2024-03-25T19:30:52.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 10 client.4707.objecter ms_dispatch 0x7f7674000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:30:52.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:30:52.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:30:52.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T19:30:52.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76640502f0 con 0x7f76740b4d60 2024-03-25T19:30:52.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:30:52.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:52.975+0000 7f766affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:30:53.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.230+0000 7f767c4e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:53.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.230+0000 7f767c4e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:53.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.230+0000 7f767c4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.60:6789/0 2024-03-25T19:30:53.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.230+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 --> v1:172.21.15.60:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76740627b0 con 0x7f76740b4d60 2024-03-25T19:30:53.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.269+0000 7f766affd640 1 -- 172.21.15.60:0/2355703432 <== mon.0 v1:172.21.15.60:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f7664052220 con 0x7f76740b4d60 2024-03-25T19:30:53.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.269+0000 7f766affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:30:53.268 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.269+0000 7f766affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:30:53.268 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:30:53.269 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 10 librados: watch_flush enter 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 10 librados: watch_flush exit 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 10 client.4707.objecter successfully canceled tick 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 >> v1:172.21.15.90:6800/1827638754 conn(0x7f765806dd90 legacy=0x7f7658070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 10 monclient: shutdown 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 >> v1:172.21.15.60:6789/0 conn(0x7f76740b4d60 legacy=0x7f767413a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:30:53.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 shutdown_connections 2024-03-25T19:30:53.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 >> 172.21.15.60:0/2355703432 conn(0x7f76740617a0 msgr2=0x7f767413c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:30:53.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 shutdown_connections 2024-03-25T19:30:53.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 1 -- 172.21.15.60:0/2355703432 wait complete. 2024-03-25T19:30:53.272 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:30:53.271+0000 7f767c4e6640 1 librados: shutdown 2024-03-25T19:30:53.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:00.090 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:00.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.202+0000 7f2ebc6d3640 10 monclient: get_monmap_and_config 2024-03-25T19:31:00.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.202+0000 7f2ebc6d3640 10 monclient: build_initial_monmap 2024-03-25T19:31:00.202 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.202+0000 7f2ebc6d3640 10 monclient: monmap: 2024-03-25T19:31:00.202 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:00.202 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:00.202 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:31:00.203421+0000 2024-03-25T19:31:00.203 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:31:00.203421+0000 2024-03-25T19:31:00.203 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:00.203 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:00.203 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:00.203 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:31:00.203 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:31:00.203 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:00.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.203+0000 7f2ebc6d3640 1 Processor -- start 2024-03-25T19:31:00.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.203+0000 7f2ebc6d3640 1 -- start start 2024-03-25T19:31:00.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.203+0000 7f2ebc6d3640 10 monclient: init 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb40705b0 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb41402d0 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb4149ae0 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb40718c0 con 0x7f2eb40705b0 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb40714c0 con 0x7f2eb41402d0 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb4072b50 con 0x7f2eb4149ae0 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:00.205 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.204+0000 7f2ebc6d3640 10 monclient(hunting): authenticate will time out at 6987.302335s 2024-03-25T19:31:00.212 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.213+0000 7f2eb9c6e640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2eb41402d0 0x7f2eb4140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:42998/0 (socket says 172.21.15.60:42998) 2024-03-25T19:31:00.213 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.213+0000 7f2eb9c6e640 1 -- 172.21.15.60:0/708396762 learned_addr learned my addr 172.21.15.60:0/708396762 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:00.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.226+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211183163 0 0) 0x7f2eb40714c0 con 0x7f2eb41402d0 2024-03-25T19:31:00.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.226+0000 7f2eb946d640 10 monclient(hunting): _negotiate 2024-03-25T19:31:00.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.226+0000 7f2eb946d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:00.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.226+0000 7f2eb946d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:00.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.226+0000 7f2eb946d640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:31:00.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.226+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea0003520 con 0x7f2eb41402d0 2024-03-25T19:31:00.241 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.242+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718824809 0 0) 0x7f2eb4072b50 con 0x7f2eb4149ae0 2024-03-25T19:31:00.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.242+0000 7f2eb946d640 10 monclient(hunting): _negotiate 2024-03-25T19:31:00.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.242+0000 7f2eb946d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:00.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.242+0000 7f2eb946d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:00.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.242+0000 7f2eb946d640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:31:00.242 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.242+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb40714c0 con 0x7f2eb4149ae0 2024-03-25T19:31:00.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652668074 0 0) 0x7f2ea0003520 con 0x7f2eb41402d0 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 >> v1:172.21.15.60:6790/0 conn(0x7f2eb4149ae0 legacy=0x7f2eb414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 >> v1:172.21.15.60:6789/0 conn(0x7f2eb40705b0 legacy=0x7f2eb413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 10 monclient: _finish_hunting 0 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 1 monclient: found mon.noname-c 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb413dd10 con 0x7f2eb41402d0 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 10 monclient: _finish_auth 0 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 10 monclient: _check_auth_tickets 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2ea4001690 con 0x7f2eb41402d0 2024-03-25T19:31:00.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2eb946d640 10 monclient: dump: 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2ebc6d3640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:31:00.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.244+0000 7f2ebc6d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ea40017f0 con 0x7f2eb41402d0 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2eb946d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2eb946d640 1 -- 172.21.15.60:0/708396762 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2ea4001c00 con 0x7f2eb41402d0 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2eb946d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2eb946d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2eb946d640 10 monclient: dump: 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.260 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2ebc6d3640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:00.261 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.260+0000 7f2ebc6d3640 4 set_mon_vals no callback set 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.261+0000 7f2ebc6d3640 10 monclient: shutdown 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.261+0000 7f2ebc6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.261+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/708396762 >> v1:172.21.15.90:6789/0 conn(0x7f2eb41402d0 legacy=0x7f2eb4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.261+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/708396762 shutdown_connections 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.261+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/708396762 >> 172.21.15.60:0/708396762 conn(0x7f2eb40b5d10 msgr2=0x7f2eb40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.261+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/708396762 shutdown_connections 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.261+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/708396762 wait complete. 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 10 monclient: build_initial_monmap 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 10 monclient: monmap: 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:00.263 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 1 librados: starting msgr at 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 1 librados: starting objecter 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 1 Processor -- start 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 1 -- start start 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 1 librados: setting wanted keys 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 1 librados: calling monclient init 2024-03-25T19:31:00.264 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.262+0000 7f2ebc6d3640 10 monclient: init 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 10 monclient(hunting): picked mon.noname-b con 0x7f2eb40705b0 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 10 monclient(hunting): picked mon.noname-c con 0x7f2eb41402d0 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 10 monclient(hunting): picked mon.noname-a con 0x7f2eb4149ae0 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb406c1b0 con 0x7f2eb4149ae0 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb406bc90 con 0x7f2eb40705b0 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb406be20 con 0x7f2eb41402d0 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:00.265 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.263+0000 7f2ebc6d3640 10 monclient(hunting): authenticate will time out at 6987.361658s 2024-03-25T19:31:00.267 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.267+0000 7f2eba46f640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f2eb40705b0 0x7f2eb4139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:43056/0 (socket says 172.21.15.60:43056) 2024-03-25T19:31:00.267 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.267+0000 7f2eba46f640 1 -- 172.21.15.60:0/3849322287 learned_addr learned my addr 172.21.15.60:0/3849322287 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:00.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.272+0000 7f2eaaffd640 10 client.?.objecter ms_handle_connect 0x7f2eb40705b0 2024-03-25T19:31:00.271 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.272+0000 7f2eaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:00.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.280+0000 7f2eaaffd640 10 client.?.objecter ms_handle_connect 0x7f2eb41402d0 2024-03-25T19:31:00.280 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.280+0000 7f2eaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:00.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.282+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352063735 0 0) 0x7f2eb406bc90 con 0x7f2eb40705b0 2024-03-25T19:31:00.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.282+0000 7f2eaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:00.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.282+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:00.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.282+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:00.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.282+0000 7f2eaaffd640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:31:00.282 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.282+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e98004eb0 con 0x7f2eb40705b0 2024-03-25T19:31:00.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.285+0000 7f2eaaffd640 10 client.?.objecter ms_handle_connect 0x7f2eb4149ae0 2024-03-25T19:31:00.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.285+0000 7f2eaaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:00.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.290+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216784559 0 0) 0x7f2eb406be20 con 0x7f2eb41402d0 2024-03-25T19:31:00.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.290+0000 7f2eaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:00.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.290+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:00.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.290+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:00.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.290+0000 7f2eaaffd640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:31:00.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.290+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb406bc90 con 0x7f2eb41402d0 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.300+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1443736531 0 0) 0x7f2eb406bc90 con 0x7f2eb41402d0 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.300+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eb406be20 con 0x7f2eb41402d0 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.300+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2ea4001b00 con 0x7f2eb41402d0 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.300+0000 7f2eaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.300+0000 7f2eaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.300+0000 7f2eaaffd640 10 monclient(hunting): dump: 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.300 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:00.301 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:00.301 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:00.301 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:00.301 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:00.301 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:00.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.300+0000 7f2eaaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:00.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.307+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018451910 0 0) 0x7f2e98004eb0 con 0x7f2eb40705b0 2024-03-25T19:31:00.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.307+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2eb406bc90 con 0x7f2eb40705b0 2024-03-25T19:31:00.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.308+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2eb0001a30 con 0x7f2eb40705b0 2024-03-25T19:31:00.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.308+0000 7f2eaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:00.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.308+0000 7f2eaaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:00.307 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.308+0000 7f2eaaffd640 10 monclient(hunting): dump: 2024-03-25T19:31:00.307 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:00.308 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:00.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.313+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889329112 0 0) 0x7f2eb406c1b0 con 0x7f2eb4149ae0 2024-03-25T19:31:00.312 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.313+0000 7f2eaaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:00.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.313+0000 7f2eaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:00.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.313+0000 7f2eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:00.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.313+0000 7f2eaaffd640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:31:00.313 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.313+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e98004eb0 con 0x7f2eb4149ae0 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4240950026 0 0) 0x7f2eb406be20 con 0x7f2eb41402d0 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 >> v1:172.21.15.90:6789/0 conn(0x7f2eb40705b0 legacy=0x7f2eb4139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 >> v1:172.21.15.60:6789/0 conn(0x7f2eb4149ae0 legacy=0x7f2eb41e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 1 monclient: found mon.c 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:00.319 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb413fc90 con 0x7f2eb41402d0 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2eaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 5 monclient: authenticate success, global_id 4694 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 10 monclient: _renew_subs 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb413fee0 con 0x7f2eb41402d0 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 10 monclient: _renew_subs 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb41db460 con 0x7f2eb41402d0 2024-03-25T19:31:00.320 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.319+0000 7f2ebc6d3640 1 librados: init done 2024-03-25T19:31:00.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.320+0000 7f2ebc6d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:00.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.320+0000 7f2ebc6d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:00.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.320+0000 7f2ebc6d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:00.321 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.320+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb40627b0 con 0x7f2eb41402d0 2024-03-25T19:31:00.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.327+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ea4001ca0 con 0x7f2eb41402d0 2024-03-25T19:31:00.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.327+0000 7f2eaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:00.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.327+0000 7f2eb8c6c640 4 set_mon_vals no callback set 2024-03-25T19:31:00.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.618+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f2ea4004c30 con 0x7f2eb41402d0 2024-03-25T19:31:00.617 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.618+0000 7f2eaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.618+0000 7f2eaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.618+0000 7f2eaaffd640 10 monclient: dump: 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:00.618 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:00.619 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:00.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.618+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f2ea401ac00 con 0x7f2eb41402d0 2024-03-25T19:31:00.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.619+0000 7f2eba46f640 10 monclient: get_auth_request con 0x7f2e9806df50 auth_method 0 2024-03-25T19:31:00.619 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.620+0000 7f2eaaffd640 10 client.4694.objecter ms_handle_connect 0x7f2e9806df50 2024-03-25T19:31:00.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.627+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f2ea4054820 con 0x7f2eb41402d0 2024-03-25T19:31:00.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.627+0000 7f2eaaffd640 10 client.4694.objecter ms_dispatch 0x7f2eb4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:00.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.627+0000 7f2eaaffd640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:00.627 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.627+0000 7f2eaaffd640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:00.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.627+0000 7f2eaaffd640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T19:31:00.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.627+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ea4050330 con 0x7f2eb41402d0 2024-03-25T19:31:00.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.628+0000 7f2eaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:00.628 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.628+0000 7f2eaaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:00.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.862+0000 7f2ebc6d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:00.861 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.862+0000 7f2ebc6d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:00.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.862+0000 7f2ebc6d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:00.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.862+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb406c360 con 0x7f2eb41402d0 2024-03-25T19:31:00.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.976+0000 7f2eaaffd640 1 -- 172.21.15.60:0/3849322287 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f2eb406c360 con 0x7f2eb41402d0 2024-03-25T19:31:00.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.976+0000 7f2eaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:00.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.976+0000 7f2eaaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:00.976 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:31:00.976 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:00.978 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 10 librados: watch_flush enter 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 10 librados: watch_flush exit 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 10 client.4694.objecter successfully canceled tick 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 >> v1:172.21.15.90:6800/1827638754 conn(0x7f2e9806df50 legacy=0x7f2e98070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 10 monclient: shutdown 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 >> v1:172.21.15.60:6790/0 conn(0x7f2eb41402d0 legacy=0x7f2eb41dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 shutdown_connections 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 >> 172.21.15.60:0/3849322287 conn(0x7f2eb40b5d10 msgr2=0x7f2eb40b3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:00.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 shutdown_connections 2024-03-25T19:31:00.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 1 -- 172.21.15.60:0/3849322287 wait complete. 2024-03-25T19:31:00.980 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:00.978+0000 7f2ebc6d3640 1 librados: shutdown 2024-03-25T19:31:00.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:01.447 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:02.158 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.459 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:02.170 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.501 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:02.212 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.505 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:02.159 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.513 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:02.167 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.516 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:02.227 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.527 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:02.238 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.544 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:02.198 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:01.555 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:02.267 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:07.796 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:07.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.912+0000 7f9f4e64a640 10 monclient: get_monmap_and_config 2024-03-25T19:31:07.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.912+0000 7f9f4e64a640 10 monclient: build_initial_monmap 2024-03-25T19:31:07.912 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.912+0000 7f9f4e64a640 10 monclient: monmap: 2024-03-25T19:31:07.912 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:07.912 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:07.912 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:31:07.913519+0000 2024-03-25T19:31:07.913 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:31:07.913519+0000 2024-03-25T19:31:07.913 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:07.913 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:07.913 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:07.913 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:31:07.913 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:31:07.913 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:07.914 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.913+0000 7f9f4e64a640 1 Processor -- start 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.913+0000 7f9f4e64a640 1 -- start start 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.913+0000 7f9f4e64a640 10 monclient: init 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.914+0000 7f9f4e64a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.914+0000 7f9f4e64a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.914+0000 7f9f4e64a640 10 monclient(hunting): picked mon.noname-a con 0x7f9f48061890 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.914+0000 7f9f4e64a640 10 monclient(hunting): picked mon.noname-b con 0x7f9f4806e740 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.914+0000 7f9f4e64a640 10 monclient(hunting): picked mon.noname-c con 0x7f9f480ae280 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.914+0000 7f9f4e64a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f480ae830 con 0x7f9f48061890 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.915+0000 7f9f4e64a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4806f720 con 0x7f9f480ae280 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.915+0000 7f9f4e64a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f480ada80 con 0x7f9f4806e740 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.915+0000 7f9f4e64a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:07.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.915+0000 7f9f4e64a640 10 monclient(hunting): authenticate will time out at 6995.013042s 2024-03-25T19:31:07.918 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.919+0000 7f9f47fff640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f9f48061890 0x7f9f4813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:53228/0 (socket says 172.21.15.60:53228) 2024-03-25T19:31:07.919 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.919+0000 7f9f47fff640 1 -- 172.21.15.60:0/1918541707 learned_addr learned my addr 172.21.15.60:0/1918541707 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:07.928 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.929+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879312427 0 0) 0x7f9f480ada80 con 0x7f9f4806e740 2024-03-25T19:31:07.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.929+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:07.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.929+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:07.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.929+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:07.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.929+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:31:07.929 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.929+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f30003520 con 0x7f9f4806e740 2024-03-25T19:31:07.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.933+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683622231 0 0) 0x7f9f480ae830 con 0x7f9f48061890 2024-03-25T19:31:07.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.933+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:07.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.933+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:07.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.933+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:07.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.933+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:31:07.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.933+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f480ada80 con 0x7f9f48061890 2024-03-25T19:31:07.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.934+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785423792 0 0) 0x7f9f4806f720 con 0x7f9f480ae280 2024-03-25T19:31:07.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.934+0000 7f9f46ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:07.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.934+0000 7f9f46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:07.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.934+0000 7f9f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:07.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.934+0000 7f9f46ffd640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:31:07.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.934+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f480ae830 con 0x7f9f480ae280 2024-03-25T19:31:07.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 676291366 0 0) 0x7f9f30003520 con 0x7f9f4806e740 2024-03-25T19:31:07.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 >> v1:172.21.15.90:6789/0 conn(0x7f9f480ae280 legacy=0x7f9f48145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:07.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 >> v1:172.21.15.60:6789/0 conn(0x7f9f48061890 legacy=0x7f9f4813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:07.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:07.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 1 monclient: found mon.noname-b 2024-03-25T19:31:07.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4813fbd0 con 0x7f9f4806e740 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f3c001690 con 0x7f9f4806e740 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f46ffd640 10 monclient: dump: 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.938 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:07.939 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:07.939 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:07.939 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:07.939 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:07.939 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:07.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f4e64a640 5 monclient: authenticate success, global_id 4697 2024-03-25T19:31:07.939 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.937+0000 7f9f4e64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:07.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f3c0017f0 con 0x7f9f4806e740 2024-03-25T19:31:07.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f46ffd640 1 -- 172.21.15.60:0/1918541707 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f3c001c00 con 0x7f9f4806e740 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f46ffd640 10 monclient: dump: 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:07.944 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:07.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f4e64a640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:07.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.944+0000 7f9f4e64a640 4 set_mon_vals no callback set 2024-03-25T19:31:07.946 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.945+0000 7f9f4e64a640 10 monclient: shutdown 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.945+0000 7f9f4e64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.945+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1918541707 >> v1:172.21.15.60:6790/0 conn(0x7f9f4806e740 legacy=0x7f9f48145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.945+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1918541707 shutdown_connections 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.945+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1918541707 >> 172.21.15.60:0/1918541707 conn(0x7f9f480b0ef0 msgr2=0x7f9f480b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.945+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1918541707 shutdown_connections 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.945+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1918541707 wait complete. 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.946+0000 7f9f4e64a640 10 monclient: build_initial_monmap 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.946+0000 7f9f4e64a640 10 monclient: monmap: 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:31:07.947 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.946+0000 7f9f4e64a640 1 librados: starting msgr at 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.946+0000 7f9f4e64a640 1 librados: starting objecter 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.946+0000 7f9f4e64a640 1 Processor -- start 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 1 -- start start 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 1 librados: setting wanted keys 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 1 librados: calling monclient init 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient: init 2024-03-25T19:31:07.948 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient(hunting): picked mon.noname-c con 0x7f9f48061890 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient(hunting): picked mon.noname-b con 0x7f9f480ae280 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient(hunting): picked mon.noname-a con 0x7f9f4813e620 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f480aa860 con 0x7f9f4813e620 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f48146610 con 0x7f9f480ae280 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f481462e0 con 0x7f9f48061890 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:07.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.947+0000 7f9f4e64a640 10 monclient(hunting): authenticate will time out at 6995.045666s 2024-03-25T19:31:07.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.956+0000 7f9f477fe640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f9f480ae280 0x7f9f480aaf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:57152/0 (socket says 172.21.15.60:57152) 2024-03-25T19:31:07.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.956+0000 7f9f477fe640 1 -- 172.21.15.60:0/1936630768 learned_addr learned my addr 172.21.15.60:0/1936630768 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:07.956 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.956+0000 7f9f44ff9640 10 client.?.objecter ms_handle_connect 0x7f9f480ae280 2024-03-25T19:31:07.957 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.957+0000 7f9f44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:07.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.964+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368989425 0 0) 0x7f9f48146610 con 0x7f9f480ae280 2024-03-25T19:31:07.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.964+0000 7f9f44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:07.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.964+0000 7f9f44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:07.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.964+0000 7f9f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:07.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.964+0000 7f9f44ff9640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:31:07.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.964+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f34004eb0 con 0x7f9f480ae280 2024-03-25T19:31:07.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.966+0000 7f9f44ff9640 10 client.?.objecter ms_handle_connect 0x7f9f48061890 2024-03-25T19:31:07.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.966+0000 7f9f44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:07.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.968+0000 7f9f44ff9640 10 client.?.objecter ms_handle_connect 0x7f9f4813e620 2024-03-25T19:31:07.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.968+0000 7f9f44ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:07.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.973+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316267670 0 0) 0x7f9f480aa860 con 0x7f9f4813e620 2024-03-25T19:31:07.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.973+0000 7f9f44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:07.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.973+0000 7f9f44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:07.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.973+0000 7f9f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:07.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.973+0000 7f9f44ff9640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:31:07.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.974+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f48146610 con 0x7f9f4813e620 2024-03-25T19:31:07.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.975+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 575030972 0 0) 0x7f9f34004eb0 con 0x7f9f480ae280 2024-03-25T19:31:07.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.975+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f480aa860 con 0x7f9f480ae280 2024-03-25T19:31:07.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.975+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f3c001bc0 con 0x7f9f480ae280 2024-03-25T19:31:07.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.975+0000 7f9f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.975+0000 7f9f44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.975+0000 7f9f44ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:07.975 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.975+0000 7f9f44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:07.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.979+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456682065 0 0) 0x7f9f481462e0 con 0x7f9f48061890 2024-03-25T19:31:07.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.980+0000 7f9f44ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:07.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.980+0000 7f9f44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:07.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.980+0000 7f9f44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:07.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.980+0000 7f9f44ff9640 10 monclient(hunting): my global_id is 4700 2024-03-25T19:31:07.979 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.980+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f34004eb0 con 0x7f9f48061890 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.998+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2337646733 0 0) 0x7f9f34004eb0 con 0x7f9f48061890 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.998+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f481462e0 con 0x7f9f48061890 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.998+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f380018f0 con 0x7f9f48061890 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.998+0000 7f9f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.998+0000 7f9f44ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:07.998+0000 7f9f44ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.998 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:07.999 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:07.999 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:07.999 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:07.999 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:07.999 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:07.999 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:08.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.009+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2935915789 0 0) 0x7f9f48146610 con 0x7f9f4813e620 2024-03-25T19:31:08.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.009+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f34004eb0 con 0x7f9f4813e620 2024-03-25T19:31:08.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.009+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f40001b50 con 0x7f9f4813e620 2024-03-25T19:31:08.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.009+0000 7f9f44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:08.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.010+0000 7f9f44ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:08.009 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.010+0000 7f9f44ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:08.009 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:08.010 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2221995941 0 0) 0x7f9f481462e0 con 0x7f9f48061890 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 >> v1:172.21.15.90:6789/0 conn(0x7f9f480ae280 legacy=0x7f9f480aaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 >> v1:172.21.15.60:6789/0 conn(0x7f9f4813e620 legacy=0x7f9f48145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 1 monclient: found mon.c 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f481ea780 con 0x7f9f48061890 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:08.012 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 5 monclient: authenticate success, global_id 4700 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 10 monclient: _renew_subs 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f481e7750 con 0x7f9f48061890 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 10 client.4700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 10 monclient: _renew_subs 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f481e7be0 con 0x7f9f48061890 2024-03-25T19:31:08.013 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.012+0000 7f9f4e64a640 1 librados: init done 2024-03-25T19:31:08.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.013+0000 7f9f4e64a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:08.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.013+0000 7f9f4e64a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:08.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.013+0000 7f9f4e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:08.014 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.013+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f48146470 con 0x7f9f48061890 2024-03-25T19:31:08.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.326+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f380047e0 con 0x7f9f48061890 2024-03-25T19:31:08.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.326+0000 7f9f44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:08.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.326+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f9f38004950 con 0x7f9f48061890 2024-03-25T19:31:08.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.326+0000 7f9f44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.326+0000 7f9f44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.326+0000 7f9f44ff9640 10 monclient: dump: 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:08.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.326+0000 7f9f467fc640 4 set_mon_vals no callback set 2024-03-25T19:31:08.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.327+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f9f3801a920 con 0x7f9f48061890 2024-03-25T19:31:08.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.328+0000 7f9f477fe640 10 monclient: get_auth_request con 0x7f9f3406e0b0 auth_method 0 2024-03-25T19:31:08.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.328+0000 7f9f44ff9640 10 client.4700.objecter ms_handle_connect 0x7f9f3406e0b0 2024-03-25T19:31:08.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.333+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f9f380545a0 con 0x7f9f48061890 2024-03-25T19:31:08.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.333+0000 7f9f44ff9640 10 client.4700.objecter ms_dispatch 0x7f9f48000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:08.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.333+0000 7f9f44ff9640 3 client.4700.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:08.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.333+0000 7f9f44ff9640 3 client.4700.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:08.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.333+0000 7f9f44ff9640 20 client.4700.objecter dump_active .. 0 homeless 2024-03-25T19:31:08.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.334+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f3804fee0 con 0x7f9f48061890 2024-03-25T19:31:08.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.334+0000 7f9f44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:08.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.334+0000 7f9f44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:08.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.593+0000 7f9f4e64a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:08.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.593+0000 7f9f4e64a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:08.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.594+0000 7f9f4e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:08.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.594+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f480627b0 con 0x7f9f48061890 2024-03-25T19:31:08.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.620+0000 7f9f44ff9640 1 -- 172.21.15.60:0/1936630768 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f9f38051fe0 con 0x7f9f48061890 2024-03-25T19:31:08.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.620+0000 7f9f44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:08.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.620+0000 7f9f44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:08.620 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:31:08.620 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 10 librados: watch_flush enter 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 10 librados: watch_flush exit 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 20 client.4700.objecter shutdown clearing up homeless session... 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 10 client.4700.objecter successfully canceled tick 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 >> v1:172.21.15.90:6800/1827638754 conn(0x7f9f3406e0b0 legacy=0x7f9f34070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 10 monclient: shutdown 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 >> v1:172.21.15.60:6790/0 conn(0x7f9f48061890 legacy=0x7f9f48144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:08.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 shutdown_connections 2024-03-25T19:31:08.624 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 >> 172.21.15.60:0/1936630768 conn(0x7f9f480b0ef0 msgr2=0x7f9f4813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:08.624 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 shutdown_connections 2024-03-25T19:31:08.624 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 1 -- 172.21.15.60:0/1936630768 wait complete. 2024-03-25T19:31:08.624 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:08.623+0000 7f9f4e64a640 1 librados: shutdown 2024-03-25T19:31:08.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:15.441 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:15.554 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.555+0000 7ff00a64e640 10 monclient: get_monmap_and_config 2024-03-25T19:31:15.554 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.555+0000 7ff00a64e640 10 monclient: build_initial_monmap 2024-03-25T19:31:15.554 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.555+0000 7ff00a64e640 10 monclient: monmap: 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:31:15.555761+0000 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:31:15.555761+0000 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:31:15.555 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:15.556 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.556+0000 7ff00a64e640 1 Processor -- start 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.556+0000 7ff00a64e640 1 -- start start 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.556+0000 7ff00a64e640 10 monclient: init 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 10 monclient(hunting): picked mon.noname-b con 0x7ff0040a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 10 monclient(hunting): picked mon.noname-a con 0x7ff00413f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 10 monclient(hunting): picked mon.noname-c con 0x7ff0041430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff004142520 con 0x7ff00413f710 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0040b28a0 con 0x7ff0041430c0 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0040b2a30 con 0x7ff0040a98f0 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:15.557 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.557+0000 7ff00a64e640 10 monclient(hunting): authenticate will time out at 7002.655213s 2024-03-25T19:31:15.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.567+0000 7ff008beb640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff0041430c0 0x7ff00414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:33602/0 (socket says 172.21.15.60:33602) 2024-03-25T19:31:15.567 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.567+0000 7ff008beb640 1 -- 172.21.15.60:0/1996948139 learned_addr learned my addr 172.21.15.60:0/1996948139 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:15.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.577+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048348374 0 0) 0x7ff0040b2a30 con 0x7ff0040a98f0 2024-03-25T19:31:15.576 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.577+0000 7ff002ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:15.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.577+0000 7ff002ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:15.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.577+0000 7ff002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:15.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.577+0000 7ff002ffd640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:31:15.577 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.577+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefec003520 con 0x7ff0040a98f0 2024-03-25T19:31:15.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.583+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274360588 0 0) 0x7ff004142520 con 0x7ff00413f710 2024-03-25T19:31:15.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.583+0000 7ff002ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:15.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.583+0000 7ff002ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:15.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.583+0000 7ff002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:15.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.583+0000 7ff002ffd640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:31:15.583 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.583+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0040b2a30 con 0x7ff00413f710 2024-03-25T19:31:15.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.586+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606087417 0 0) 0x7ff0040b28a0 con 0x7ff0041430c0 2024-03-25T19:31:15.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.586+0000 7ff002ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:15.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.586+0000 7ff002ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:15.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.586+0000 7ff002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:15.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.586+0000 7ff002ffd640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:31:15.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.586+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff004142520 con 0x7ff0041430c0 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3544628060 0 0) 0x7ff004142520 con 0x7ff0041430c0 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 >> v1:172.21.15.60:6790/0 conn(0x7ff0040a98f0 legacy=0x7ff0040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 >> v1:172.21.15.60:6789/0 conn(0x7ff00413f710 legacy=0x7ff0041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 1 monclient: found mon.noname-c 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0040b44c0 con 0x7ff0041430c0 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:15.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7feffc001930 con 0x7ff0041430c0 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff002ffd640 10 monclient: dump: 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:15.594 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:15.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff00a64e640 5 monclient: authenticate success, global_id 4708 2024-03-25T19:31:15.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.593+0000 7ff00a64e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:15.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feffc001a90 con 0x7ff0041430c0 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff002ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff002ffd640 1 -- 172.21.15.60:0/1996948139 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7feffc001ee0 con 0x7ff0041430c0 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff002ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff002ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff002ffd640 10 monclient: dump: 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:15.608 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:15.609 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:15.609 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:15.609 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:15.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff00a64e640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:15.609 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.608+0000 7ff00a64e640 4 set_mon_vals no callback set 2024-03-25T19:31:15.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.609+0000 7ff00a64e640 10 monclient: shutdown 2024-03-25T19:31:15.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.609+0000 7ff00a64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:15.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.609+0000 7ff00a64e640 1 -- 172.21.15.60:0/1996948139 >> v1:172.21.15.90:6789/0 conn(0x7ff0041430c0 legacy=0x7ff00414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:15.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.609+0000 7ff00a64e640 1 -- 172.21.15.60:0/1996948139 shutdown_connections 2024-03-25T19:31:15.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.609+0000 7ff00a64e640 1 -- 172.21.15.60:0/1996948139 >> 172.21.15.60:0/1996948139 conn(0x7ff0040b7ea0 msgr2=0x7ff0040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:15.611 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.609+0000 7ff00a64e640 1 -- 172.21.15.60:0/1996948139 shutdown_connections 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.609+0000 7ff00a64e640 1 -- 172.21.15.60:0/1996948139 wait complete. 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 10 monclient: build_initial_monmap 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 10 monclient: monmap: 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:15.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 1 librados: starting msgr at 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 1 librados: starting objecter 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 1 Processor -- start 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 1 -- start start 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 1 librados: setting wanted keys 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 1 librados: calling monclient init 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.610+0000 7ff00a64e640 10 monclient: init 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 10 monclient(hunting): picked mon.noname-b con 0x7ff0040a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 10 monclient(hunting): picked mon.noname-c con 0x7ff00413f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 10 monclient(hunting): picked mon.noname-a con 0x7ff0041430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff004132f60 con 0x7ff0041430c0 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff004132cc0 con 0x7ff0040a98f0 2024-03-25T19:31:15.613 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00414df40 con 0x7ff00413f710 2024-03-25T19:31:15.614 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:15.614 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.611+0000 7ff00a64e640 10 monclient(hunting): authenticate will time out at 7002.709789s 2024-03-25T19:31:15.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.620+0000 7ff003fff640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7ff0040a98f0 0x7ff00414e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:33646/0 (socket says 172.21.15.60:33646) 2024-03-25T19:31:15.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.620+0000 7ff003fff640 1 -- 172.21.15.60:0/3157176737 learned_addr learned my addr 172.21.15.60:0/3157176737 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:15.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.620+0000 7ff000ff9640 10 client.?.objecter ms_handle_connect 0x7ff0040a98f0 2024-03-25T19:31:15.620 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.620+0000 7ff000ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:15.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.624+0000 7ff000ff9640 10 client.?.objecter ms_handle_connect 0x7ff00413f710 2024-03-25T19:31:15.623 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.624+0000 7ff000ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:15.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.679+0000 7ff000ff9640 10 client.?.objecter ms_handle_connect 0x7ff0041430c0 2024-03-25T19:31:15.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.679+0000 7ff000ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:15.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.683+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000917002 0 0) 0x7ff004132cc0 con 0x7ff0040a98f0 2024-03-25T19:31:15.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.684+0000 7ff000ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:15.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.684+0000 7ff000ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:15.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.684+0000 7ff000ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:15.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.684+0000 7ff000ff9640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:31:15.683 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.684+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff0004eb0 con 0x7ff0040a98f0 2024-03-25T19:31:15.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.700+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663462686 0 0) 0x7ff0040b44c0 con 0x7ff0041430c0 2024-03-25T19:31:15.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.700+0000 7ff000ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:15.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.700+0000 7ff000ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:15.699 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.700+0000 7ff000ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:15.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.700+0000 7ff000ff9640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:31:15.700 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.700+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff004132cc0 con 0x7ff0041430c0 2024-03-25T19:31:15.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.710+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 431295696 0 0) 0x7ff004132cc0 con 0x7ff0041430c0 2024-03-25T19:31:15.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.710+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0040b44c0 con 0x7ff0041430c0 2024-03-25T19:31:15.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.710+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7feffc001c00 con 0x7ff0041430c0 2024-03-25T19:31:15.709 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.710+0000 7ff000ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.710+0000 7ff000ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.710+0000 7ff000ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:15.710 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.710+0000 7ff000ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:15.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.714+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956330049 0 0) 0x7feff0004eb0 con 0x7ff0040a98f0 2024-03-25T19:31:15.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.714+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff004132cc0 con 0x7ff0040a98f0 2024-03-25T19:31:15.713 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.714+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7feff40019b0 con 0x7ff0040a98f0 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.714+0000 7ff000ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.714+0000 7ff000ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.714+0000 7ff000ff9640 10 monclient(hunting): dump: 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:15.714 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:15.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2271738210 0 0) 0x7ff004132cc0 con 0x7ff0040a98f0 2024-03-25T19:31:15.728 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 >> v1:172.21.15.60:6790/0 conn(0x7ff00413f710 legacy=0x7ff00414bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 >> v1:172.21.15.60:6789/0 conn(0x7ff0041430c0 legacy=0x7ff004139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 1 monclient: found mon.b 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00414d560 con 0x7ff0040a98f0 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff000ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 10 monclient: _renew_subs 2024-03-25T19:31:15.729 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:15.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00414d840 con 0x7ff0040a98f0 2024-03-25T19:31:15.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:15.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 10 monclient: _renew_subs 2024-03-25T19:31:15.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:15.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.728+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0040b7bd0 con 0x7ff0040a98f0 2024-03-25T19:31:15.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.729+0000 7ff00a64e640 1 librados: init done 2024-03-25T19:31:15.730 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.730+0000 7ff00a64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:15.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.730+0000 7ff00a64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:15.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.730+0000 7ff00a64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:15.731 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:15.730+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0040b3ef0 con 0x7ff0040a98f0 2024-03-25T19:31:16.019 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.020+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feff40035f0 con 0x7ff0040a98f0 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.020+0000 7ff000ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.020+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7feff4003980 con 0x7ff0040a98f0 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.020+0000 7ff000ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.020+0000 7ff000ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.020+0000 7ff000ff9640 10 monclient: dump: 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:16.020 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:16.021 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:16.021 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:16.021 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:16.021 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:16.021 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:16.021 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:16.021 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.020+0000 7ff0027fc640 4 set_mon_vals no callback set 2024-03-25T19:31:16.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.021+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7feff4003b60 con 0x7ff0040a98f0 2024-03-25T19:31:16.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.021+0000 7ff0037fe640 10 monclient: get_auth_request con 0x7feff006de90 auth_method 0 2024-03-25T19:31:16.022 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.022+0000 7ff000ff9640 10 client.4711.objecter ms_handle_connect 0x7feff006de90 2024-03-25T19:31:16.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7feff4054830 con 0x7ff0040a98f0 2024-03-25T19:31:16.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 10 client.4711.objecter ms_dispatch 0x7ff004000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:16.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:16.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:16.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:31:16.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feff4050340 con 0x7ff0040a98f0 2024-03-25T19:31:16.026 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:16.027 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.026+0000 7ff000ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:16.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.289+0000 7ff00a64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:16.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.289+0000 7ff00a64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:16.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.289+0000 7ff00a64e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:16.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.289+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0040aae40 con 0x7ff0040a98f0 2024-03-25T19:31:16.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.326+0000 7ff000ff9640 1 -- 172.21.15.60:0/3157176737 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7feff4052270 con 0x7ff0040a98f0 2024-03-25T19:31:16.325 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.326+0000 7ff000ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:16.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.326+0000 7ff000ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:16.326 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:31:16.326 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.327+0000 7ff00a64e640 10 librados: watch_flush enter 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.327+0000 7ff00a64e640 10 librados: watch_flush exit 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.327+0000 7ff00a64e640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.327+0000 7ff00a64e640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 >> v1:172.21.15.90:6800/1827638754 conn(0x7feff006de90 legacy=0x7feff0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 10 monclient: shutdown 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 >> v1:172.21.15.90:6789/0 conn(0x7ff0040a98f0 legacy=0x7ff00414e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 shutdown_connections 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 >> 172.21.15.60:0/3157176737 conn(0x7ff0040b7ea0 msgr2=0x7ff004074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 shutdown_connections 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 1 -- 172.21.15.60:0/3157176737 wait complete. 2024-03-25T19:31:16.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:16.328+0000 7ff00a64e640 1 librados: shutdown 2024-03-25T19:31:16.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:21.494 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:22.205 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.503 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:22.214 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.548 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:22.260 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.559 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:22.213 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.563 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:22.274 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.568 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:22.222 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.570 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:22.281 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.586 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:22.241 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:21.592 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:22.304 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:23.147 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:23.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.257+0000 7efd53313640 10 monclient: get_monmap_and_config 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.257+0000 7efd53313640 10 monclient: build_initial_monmap 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.257+0000 7efd53313640 10 monclient: monmap: 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:31:23.259097+0000 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:31:23.259097+0000 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:31:23.258 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.258+0000 7efd53313640 1 Processor -- start 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.258+0000 7efd53313640 1 -- start start 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.258+0000 7efd53313640 10 monclient: init 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 10 monclient(hunting): picked mon.noname-b con 0x7efd4c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 10 monclient(hunting): picked mon.noname-c con 0x7efd4c0b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 10 monclient(hunting): picked mon.noname-a con 0x7efd4c14be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c0af940 con 0x7efd4c14be90 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c0b0a70 con 0x7efd4c0b1730 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c0b0d50 con 0x7efd4c0b4d60 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:23.260 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.259+0000 7efd53313640 10 monclient(hunting): authenticate will time out at 7010.358224s 2024-03-25T19:31:23.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.269+0000 7efd510af640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7efd4c0b4d60 0x7efd4c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:45104/0 (socket says 172.21.15.60:45104) 2024-03-25T19:31:23.269 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.269+0000 7efd510af640 1 -- 172.21.15.60:0/3638924829 learned_addr learned my addr 172.21.15.60:0/3638924829 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:23.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.278+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857698722 0 0) 0x7efd4c0b0d50 con 0x7efd4c0b4d60 2024-03-25T19:31:23.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.278+0000 7efd43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:23.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.278+0000 7efd43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:23.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.278+0000 7efd43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:23.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.278+0000 7efd43fff640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:31:23.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.278+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd38003520 con 0x7efd4c0b4d60 2024-03-25T19:31:23.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.284+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090821081 0 0) 0x7efd4c0b0a70 con 0x7efd4c0b1730 2024-03-25T19:31:23.284 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.284+0000 7efd43fff640 10 monclient(hunting): _negotiate 2024-03-25T19:31:23.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.284+0000 7efd43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:23.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.284+0000 7efd43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:23.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.284+0000 7efd43fff640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:31:23.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.284+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c0b0d50 con 0x7efd4c0b1730 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 869982663 0 0) 0x7efd38003520 con 0x7efd4c0b4d60 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 >> v1:172.21.15.90:6789/0 conn(0x7efd4c0b1730 legacy=0x7efd4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 >> v1:172.21.15.60:6789/0 conn(0x7efd4c14be90 legacy=0x7efd4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 10 monclient: _finish_hunting 0 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 1 monclient: found mon.noname-b 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.60:6790/0 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c0b1030 con 0x7efd4c0b4d60 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 10 monclient: _finish_auth 0 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 10 monclient: _check_auth_tickets 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7efd48001690 con 0x7efd4c0b4d60 2024-03-25T19:31:23.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd53313640 5 monclient: authenticate success, global_id 4709 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd53313640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd43fff640 10 monclient: dump: 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:23.289 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:23.290 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.287+0000 7efd53313640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:23.300 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 <== mon.2 v1:172.21.15.60:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd480017f0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.326 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd43fff640 1 -- 172.21.15.60:0/3638924829 <== mon.2 v1:172.21.15.60:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7efd48001c00 con 0x7efd4c0b4d60 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd43fff640 10 monclient: dump: 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:23.327 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:23.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd53313640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:23.328 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.299+0000 7efd53313640 4 set_mon_vals no callback set 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.300+0000 7efd53313640 10 monclient: shutdown 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.300+0000 7efd53313640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.300+0000 7efd53313640 1 -- 172.21.15.60:0/3638924829 >> v1:172.21.15.60:6790/0 conn(0x7efd4c0b4d60 legacy=0x7efd4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.300+0000 7efd53313640 1 -- 172.21.15.60:0/3638924829 shutdown_connections 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.300+0000 7efd53313640 1 -- 172.21.15.60:0/3638924829 >> 172.21.15.60:0/3638924829 conn(0x7efd4c0617a0 msgr2=0x7efd4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.300+0000 7efd53313640 1 -- 172.21.15.60:0/3638924829 shutdown_connections 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.300+0000 7efd53313640 1 -- 172.21.15.60:0/3638924829 wait complete. 2024-03-25T19:31:23.330 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.301+0000 7efd53313640 10 monclient: build_initial_monmap 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.301+0000 7efd53313640 10 monclient: monmap: 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 librados: starting msgr at 2024-03-25T19:31:23.331 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 librados: starting objecter 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 Processor -- start 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 -- start start 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 librados: setting wanted keys 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 librados: calling monclient init 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient: init 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient(hunting): picked mon.noname-a con 0x7efd4c0b1730 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient(hunting): picked mon.noname-c con 0x7efd4c0b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient(hunting): picked mon.noname-b con 0x7efd4c14be90 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c138010 con 0x7efd4c0b1730 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c1382f0 con 0x7efd4c14be90 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd4c136d90 con 0x7efd4c0b4d60 2024-03-25T19:31:23.332 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.302+0000 7efd53313640 10 monclient(hunting): authenticate will time out at 7010.401679s 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.312+0000 7efd510af640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7efd4c0b1730 0x7efd4c135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:38872/0 (socket says 172.21.15.60:38872) 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.312+0000 7efd510af640 1 -- 172.21.15.60:0/1177664244 learned_addr learned my addr 172.21.15.60:0/1177664244 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.313+0000 7efd41ffb640 10 client.?.objecter ms_handle_connect 0x7efd4c0b1730 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.313+0000 7efd41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.316+0000 7efd41ffb640 10 client.?.objecter ms_handle_connect 0x7efd4c0b4d60 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.316+0000 7efd41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.321+0000 7efd41ffb640 10 client.?.objecter ms_handle_connect 0x7efd4c14be90 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.321+0000 7efd41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.326+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533249299 0 0) 0x7efd4c138010 con 0x7efd4c0b1730 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.326+0000 7efd41ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.326+0000 7efd41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.326+0000 7efd41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:23.333 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.326+0000 7efd41ffb640 10 monclient(hunting): my global_id is 4731 2024-03-25T19:31:23.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.326+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd30004eb0 con 0x7efd4c0b1730 2024-03-25T19:31:23.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.331+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803337056 0 0) 0x7efd4c136d90 con 0x7efd4c0b4d60 2024-03-25T19:31:23.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.331+0000 7efd41ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:23.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.331+0000 7efd41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:23.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.331+0000 7efd41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:23.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.331+0000 7efd41ffb640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:31:23.334 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.331+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c138010 con 0x7efd4c0b4d60 2024-03-25T19:31:23.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.341+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 282074384 0 0) 0x7efd4c138010 con 0x7efd4c0b4d60 2024-03-25T19:31:23.341 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.341+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd4c136d90 con 0x7efd4c0b4d60 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.341+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7efd3c001930 con 0x7efd4c0b4d60 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.341+0000 7efd41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.341+0000 7efd41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.341+0000 7efd41ffb640 10 monclient(hunting): dump: 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:23.342 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:23.343 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:23.343 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:23.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.341+0000 7efd41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:23.343 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.343+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733838465 0 0) 0x7efd4c1382f0 con 0x7efd4c14be90 2024-03-25T19:31:23.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.343+0000 7efd41ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:23.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.343+0000 7efd41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:23.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.343+0000 7efd41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:23.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.343+0000 7efd41ffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:31:23.344 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.343+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd4c138010 con 0x7efd4c14be90 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2543380818 0 0) 0x7efd4c136d90 con 0x7efd4c0b4d60 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 >> v1:172.21.15.90:6789/0 conn(0x7efd4c14be90 legacy=0x7efd4c1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 >> v1:172.21.15.60:6789/0 conn(0x7efd4c0b1730 legacy=0x7efd4c135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 1 monclient: found mon.c 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:23.353 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4c144010 con 0x7efd4c0b4d60 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 5 monclient: authenticate success, global_id 4712 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 10 monclient: _renew_subs 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4c1372b0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 10 client.4712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 10 monclient: _renew_subs 2024-03-25T19:31:23.354 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:23.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4c1369c0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.352+0000 7efd53313640 1 librados: init done 2024-03-25T19:31:23.355 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.353+0000 7efd53313640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:23.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.353+0000 7efd53313640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:23.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.353+0000 7efd53313640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:23.356 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.353+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd4c136f20 con 0x7efd4c0b4d60 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.676+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd3c003610 con 0x7efd4c0b4d60 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.676+0000 7efd41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.676+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7efd3c0039a0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.676+0000 7efd41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.676+0000 7efd41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.676+0000 7efd41ffb640 10 monclient: dump: 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:23.677 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:23.678 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.676+0000 7efd437fe640 4 set_mon_vals no callback set 2024-03-25T19:31:23.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.677+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7efd3c003bc0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.677+0000 7efd510af640 10 monclient: get_auth_request con 0x7efd3006de10 auth_method 0 2024-03-25T19:31:23.679 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.678+0000 7efd41ffb640 10 client.4712.objecter ms_handle_connect 0x7efd3006de10 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7efd3c053590 con 0x7efd4c0b4d60 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 10 client.4712.objecter ms_dispatch 0x7efd4c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 3 client.4712.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 3 client.4712.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 20 client.4712.objecter dump_active .. 0 homeless 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd3c04f0a0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:23.685 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.684+0000 7efd41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.921+0000 7efd53313640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.921+0000 7efd53313640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.921+0000 7efd53313640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:23.922 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.921+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd4c1441a0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.950+0000 7efd41ffb640 1 -- 172.21.15.60:0/1177664244 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7efd3c050fd0 con 0x7efd4c0b4d60 2024-03-25T19:31:23.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.950+0000 7efd41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:23.951 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.950+0000 7efd41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:23.951 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:31:23.951 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.952+0000 7efd53313640 10 librados: watch_flush enter 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.952+0000 7efd53313640 10 librados: watch_flush exit 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.952+0000 7efd53313640 20 client.4712.objecter shutdown clearing up homeless session... 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.952+0000 7efd53313640 10 client.4712.objecter successfully canceled tick 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 >> v1:172.21.15.90:6800/1827638754 conn(0x7efd3006de10 legacy=0x7efd300702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 10 monclient: shutdown 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 >> v1:172.21.15.60:6790/0 conn(0x7efd4c0b4d60 legacy=0x7efd4c143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:23.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 shutdown_connections 2024-03-25T19:31:23.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 >> 172.21.15.60:0/1177664244 conn(0x7efd4c0617a0 msgr2=0x7efd4c0b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:23.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 shutdown_connections 2024-03-25T19:31:23.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 1 -- 172.21.15.60:0/1177664244 wait complete. 2024-03-25T19:31:23.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:23.953+0000 7efd53313640 1 librados: shutdown 2024-03-25T19:31:23.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:26.171 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:26.882 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.177 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:26.889 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.189 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:26.843 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.191 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:26.903 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.192 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:26.846 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.194 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:26.906 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.210 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:26.864 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.251 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:26.962 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:26.265 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:26.976 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:30.771 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.885+0000 7f528959a640 10 monclient: get_monmap_and_config 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.885+0000 7f528959a640 10 monclient: build_initial_monmap 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.885+0000 7f528959a640 10 monclient: monmap: 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:31:30.887229+0000 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:31:30.887229+0000 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:31:30.886 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:31:30.887 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.886+0000 7f528959a640 1 Processor -- start 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.886+0000 7f528959a640 1 -- start start 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.886+0000 7f528959a640 10 monclient: init 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.886+0000 7f528959a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 10 monclient(hunting): picked mon.noname-a con 0x7f52840a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 10 monclient(hunting): picked mon.noname-c con 0x7f528413f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 10 monclient(hunting): picked mon.noname-b con 0x7f52841430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5284142520 con 0x7f52840a98f0 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52840b28a0 con 0x7f528413f710 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52840b2a30 con 0x7f52841430c0 2024-03-25T19:31:30.888 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:30.889 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.887+0000 7f528959a640 10 monclient(hunting): authenticate will time out at 7017.986063s 2024-03-25T19:31:30.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.896+0000 7f52827fc640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f528413f710 0x7f52841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:48440/0 (socket says 172.21.15.60:48440) 2024-03-25T19:31:30.896 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.896+0000 7f52827fc640 1 -- 172.21.15.60:0/3086720340 learned_addr learned my addr 172.21.15.60:0/3086720340 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:30.910 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.909+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284726714 0 0) 0x7f52840b28a0 con 0x7f528413f710 2024-03-25T19:31:30.910 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.909+0000 7f5281ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:31:30.910 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.909+0000 7f5281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:30.910 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.909+0000 7f5281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:30.910 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.909+0000 7f5281ffb640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:31:30.910 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.909+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526c003520 con 0x7f528413f710 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2427590812 0 0) 0x7f526c003520 con 0x7f528413f710 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 >> v1:172.21.15.60:6790/0 conn(0x7f52841430c0 legacy=0x7f528414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 >> v1:172.21.15.60:6789/0 conn(0x7f52840a98f0 legacy=0x7f52840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 1 monclient: found mon.noname-c 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.90:6789/0 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52840b44c0 con 0x7f528413f710 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 10 monclient: _finish_auth 0 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:31:30.915 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5278001690 con 0x7f528413f710 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f5281ffb640 10 monclient: dump: 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:30.916 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:30.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f528959a640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:31:30.917 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.914+0000 7f528959a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:30.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 <== mon.1 v1:172.21.15.90:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52780017f0 con 0x7f528413f710 2024-03-25T19:31:30.930 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f5281ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f5281ffb640 1 -- 172.21.15.60:0/3086720340 <== mon.1 v1:172.21.15.90:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5278001c40 con 0x7f528413f710 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f5281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f5281ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f5281ffb640 10 monclient: dump: 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:30.931 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:30.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f528959a640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:30.932 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.930+0000 7f528959a640 4 set_mon_vals no callback set 2024-03-25T19:31:30.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.931+0000 7f528959a640 10 monclient: shutdown 2024-03-25T19:31:30.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.931+0000 7f528959a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:30.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.931+0000 7f528959a640 1 -- 172.21.15.60:0/3086720340 >> v1:172.21.15.90:6789/0 conn(0x7f528413f710 legacy=0x7f52841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:30.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.931+0000 7f528959a640 1 -- 172.21.15.60:0/3086720340 shutdown_connections 2024-03-25T19:31:30.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.931+0000 7f528959a640 1 -- 172.21.15.60:0/3086720340 >> 172.21.15.60:0/3086720340 conn(0x7f52840b7ea0 msgr2=0x7f52840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:30.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.931+0000 7f528959a640 1 -- 172.21.15.60:0/3086720340 shutdown_connections 2024-03-25T19:31:30.933 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.931+0000 7f528959a640 1 -- 172.21.15.60:0/3086720340 wait complete. 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 10 monclient: build_initial_monmap 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 10 monclient: monmap: 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:30.934 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 1 librados: starting msgr at 2024-03-25T19:31:30.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 1 librados: starting objecter 2024-03-25T19:31:30.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 1 Processor -- start 2024-03-25T19:31:30.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 1 -- start start 2024-03-25T19:31:30.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 1 librados: setting wanted keys 2024-03-25T19:31:30.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 1 librados: calling monclient init 2024-03-25T19:31:30.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.932+0000 7f528959a640 10 monclient: init 2024-03-25T19:31:30.935 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 10 monclient(hunting): picked mon.noname-a con 0x7f52840a98f0 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 10 monclient(hunting): picked mon.noname-c con 0x7f528413f710 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 10 monclient(hunting): picked mon.noname-b con 0x7f52841430c0 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528406fe20 con 0x7f52840a98f0 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5284070100 con 0x7f52841430c0 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528406e820 con 0x7f528413f710 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.933+0000 7f528959a640 10 monclient(hunting): authenticate will time out at 7018.032545s 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.935+0000 7f52827fc640 1 --1- >> v1:172.21.15.60:6790/0 conn(0x7f528413f710 0x7f528414c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6790/0 says I am v1:172.21.15.60:53874/0 (socket says 172.21.15.60:53874) 2024-03-25T19:31:30.936 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.935+0000 7f52827fc640 1 -- 172.21.15.60:0/3265145531 learned_addr learned my addr 172.21.15.60:0/3265145531 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:30.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.936+0000 7f526b7fe640 10 client.?.objecter ms_handle_connect 0x7f528413f710 2024-03-25T19:31:30.937 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.936+0000 7f526b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:30.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.953+0000 7f526b7fe640 10 client.?.objecter ms_handle_connect 0x7f52840a98f0 2024-03-25T19:31:30.953 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.953+0000 7f526b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:30.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.953+0000 7f526b7fe640 10 client.?.objecter ms_handle_connect 0x7f52841430c0 2024-03-25T19:31:30.954 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.953+0000 7f526b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:30.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.954+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177182726 0 0) 0x7f528406e820 con 0x7f528413f710 2024-03-25T19:31:30.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.954+0000 7f526b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:30.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.954+0000 7f526b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:30.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.954+0000 7f526b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:30.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.954+0000 7f526b7fe640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:31:30.955 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.954+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5270004eb0 con 0x7f528413f710 2024-03-25T19:31:30.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.961+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113918515 0 0) 0x7f5284070100 con 0x7f52841430c0 2024-03-25T19:31:30.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.961+0000 7f526b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:30.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.961+0000 7f526b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:30.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.961+0000 7f526b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:30.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.961+0000 7f526b7fe640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:31:30.962 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.961+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528406e820 con 0x7f52841430c0 2024-03-25T19:31:30.964 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.964+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3597318434 0 0) 0x7f5270004eb0 con 0x7f528413f710 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.964+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5284070100 con 0x7f528413f710 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.964+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5278001bc0 con 0x7f528413f710 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.964+0000 7f526b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.964+0000 7f526b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.964+0000 7f526b7fe640 10 monclient(hunting): dump: 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:30.965 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.964+0000 7f526b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.966+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152465137 0 0) 0x7f528406fe20 con 0x7f52840a98f0 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.966+0000 7f526b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.966+0000 7f526b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.966+0000 7f526b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:30.966 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.966+0000 7f526b7fe640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:31:30.967 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.966+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5270004eb0 con 0x7f52840a98f0 2024-03-25T19:31:30.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.973+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2107687092 0 0) 0x7f528406e820 con 0x7f52841430c0 2024-03-25T19:31:30.973 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.973+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f528406fe20 con 0x7f52841430c0 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.973+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f527c001b10 con 0x7f52841430c0 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.973+0000 7f526b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.973+0000 7f526b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.973+0000 7f526b7fe640 10 monclient(hunting): dump: 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:30.974 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4089395608 0 0) 0x7f5284070100 con 0x7f528413f710 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 >> v1:172.21.15.90:6789/0 conn(0x7f52841430c0 legacy=0x7f528414ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 >> v1:172.21.15.60:6789/0 conn(0x7f52840a98f0 legacy=0x7f528406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 1 monclient: found mon.c 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528406e9b0 con 0x7f528413f710 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f526b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 5 monclient: authenticate success, global_id 4718 2024-03-25T19:31:30.976 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 10 monclient: _renew_subs 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52841e7700 con 0x7f528413f710 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 10 client.4718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 10 monclient: _renew_subs 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52841e7a70 con 0x7f528413f710 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.974+0000 7f528959a640 1 librados: init done 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.975+0000 7f528959a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.975+0000 7f528959a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.975+0000 7f528959a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:30.977 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:30.975+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52840627b0 con 0x7f528413f710 2024-03-25T19:31:31.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.292+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5278004bf0 con 0x7f528413f710 2024-03-25T19:31:31.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.292+0000 7f526b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:31.292 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.292+0000 7f52817fa640 4 set_mon_vals no callback set 2024-03-25T19:31:31.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.292+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f5278004f00 con 0x7f528413f710 2024-03-25T19:31:31.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.292+0000 7f526b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:31.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.292+0000 7f526b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:31.293 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.292+0000 7f526b7fe640 10 monclient: dump: 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:31.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.293+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f527801aed0 con 0x7f528413f710 2024-03-25T19:31:31.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.293+0000 7f5282ffd640 10 monclient: get_auth_request con 0x7f527006df70 auth_method 0 2024-03-25T19:31:31.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.294+0000 7f526b7fe640 10 client.4718.objecter ms_handle_connect 0x7f527006df70 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f52780549d0 con 0x7f528413f710 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 10 client.4718.objecter ms_dispatch 0x7f5284000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 3 client.4718.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 3 client.4718.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 20 client.4718.objecter dump_active .. 0 homeless 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52780504e0 con 0x7f528413f710 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:31.301 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.300+0000 7f526b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:31.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.554+0000 7f528959a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:31.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.554+0000 7f528959a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:31.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.554+0000 7f528959a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.60:6790/0 2024-03-25T19:31:31.555 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.554+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 --> v1:172.21.15.60:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52841434a0 con 0x7f528413f710 2024-03-25T19:31:31.584 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.584+0000 7f526b7fe640 1 -- 172.21.15.60:0/3265145531 <== mon.2 v1:172.21.15.60:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f5278052410 con 0x7f528413f710 2024-03-25T19:31:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.584+0000 7f526b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:31.585 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.584+0000 7f526b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:31.585 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:31:31.585 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.586+0000 7f528959a640 10 librados: watch_flush enter 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.586+0000 7f528959a640 10 librados: watch_flush exit 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.586+0000 7f528959a640 20 client.4718.objecter shutdown clearing up homeless session... 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.586+0000 7f528959a640 10 client.4718.objecter successfully canceled tick 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.586+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 >> v1:172.21.15.90:6800/1827638754 conn(0x7f527006df70 legacy=0x7f5270070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.586+0000 7f528959a640 10 monclient: shutdown 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.587+0000 7f528959a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.587+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 >> v1:172.21.15.60:6790/0 conn(0x7f528413f710 legacy=0x7f528414c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.587+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 shutdown_connections 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.587+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 >> 172.21.15.60:0/3265145531 conn(0x7f52840b7ea0 msgr2=0x7f5284074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.587+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 shutdown_connections 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.587+0000 7f528959a640 1 -- 172.21.15.60:0/3265145531 wait complete. 2024-03-25T19:31:31.588 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:31.587+0000 7f528959a640 1 librados: shutdown 2024-03-25T19:31:31.598 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:38.404 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:38.512 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.512+0000 7f4ed5c27640 10 monclient: get_monmap_and_config 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.512+0000 7f4ed5c27640 10 monclient: build_initial_monmap 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.512+0000 7f4ed5c27640 10 monclient: monmap: 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:31:38.514070+0000 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:31:38.514070+0000 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:31:38.513 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 1 Processor -- start 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 1 -- start start 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient: init 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed00b4d60 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed00b1730 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00af940 con 0x7f4ed014be90 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00b0a70 con 0x7f4ed00b1730 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00b0d50 con 0x7f4ed00b4d60 2024-03-25T19:31:38.515 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:38.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.513+0000 7f4ed5c27640 10 monclient(hunting): authenticate will time out at 7025.612681s 2024-03-25T19:31:38.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.514+0000 7f4ec6ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4ed00b1730 0x7f4ed00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:50164/0 (socket says 172.21.15.60:50164) 2024-03-25T19:31:38.516 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.514+0000 7f4ec6ffd640 1 -- 172.21.15.60:0/3873942188 learned_addr learned my addr 172.21.15.60:0/3873942188 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:38.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.580+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488867288 0 0) 0x7f4ed00af940 con 0x7f4ed014be90 2024-03-25T19:31:38.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.580+0000 7f4eceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:38.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.580+0000 7f4eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:38.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.580+0000 7f4eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:38.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.580+0000 7f4eceffd640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:31:38.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.580+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ebc003520 con 0x7f4ed014be90 2024-03-25T19:31:38.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.593+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536079948 0 0) 0x7f4ed00b0a70 con 0x7f4ed00b1730 2024-03-25T19:31:38.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.593+0000 7f4eceffd640 10 monclient(hunting): _negotiate 2024-03-25T19:31:38.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.593+0000 7f4eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:38.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.593+0000 7f4eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:38.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.593+0000 7f4eceffd640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:31:38.593 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.593+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed00af940 con 0x7f4ed00b1730 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944452807 0 0) 0x7f4ebc003520 con 0x7f4ed014be90 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 >> v1:172.21.15.60:6790/0 conn(0x7f4ed00b4d60 legacy=0x7f4ed013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 >> v1:172.21.15.90:6789/0 conn(0x7f4ed00b1730 legacy=0x7f4ed00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 10 monclient: _finish_hunting 0 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 1 monclient: found mon.noname-a 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00b1030 con 0x7f4ed014be90 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 10 monclient: _finish_auth 0 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 10 monclient: _check_auth_tickets 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:38.595 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4ed5c27640 5 monclient: authenticate success, global_id 4740 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4ec8001a10 con 0x7f4ed014be90 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4eceffd640 10 monclient: dump: 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:38.596 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.594+0000 7f4ed5c27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ec8001b70 con 0x7f4ed014be90 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4eceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4eceffd640 1 -- 172.21.15.60:0/3873942188 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4ec8001f80 con 0x7f4ed014be90 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4ed5c27640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4ed5c27640 4 set_mon_vals no callback set 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4eceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.602+0000 7f4eceffd640 10 monclient: dump: 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:38.603 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:38.604 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:38.604 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:38.604 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:38.604 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.603+0000 7f4ed5c27640 10 monclient: shutdown 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.603+0000 7f4ed5c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.603+0000 7f4ed5c27640 1 -- 172.21.15.60:0/3873942188 >> v1:172.21.15.60:6789/0 conn(0x7f4ed014be90 legacy=0x7f4ed014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.603+0000 7f4ed5c27640 1 -- 172.21.15.60:0/3873942188 shutdown_connections 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.603+0000 7f4ed5c27640 1 -- 172.21.15.60:0/3873942188 >> 172.21.15.60:0/3873942188 conn(0x7f4ed00617a0 msgr2=0x7f4ed0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.603+0000 7f4ed5c27640 1 -- 172.21.15.60:0/3873942188 shutdown_connections 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.603+0000 7f4ed5c27640 1 -- 172.21.15.60:0/3873942188 wait complete. 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.604+0000 7f4ed5c27640 10 monclient: build_initial_monmap 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.604+0000 7f4ed5c27640 10 monclient: monmap: 2024-03-25T19:31:38.606 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.604+0000 7f4ed5c27640 1 librados: starting msgr at 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.604+0000 7f4ed5c27640 1 librados: starting objecter 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.604+0000 7f4ed5c27640 1 Processor -- start 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 1 -- start start 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 1 librados: setting wanted keys 2024-03-25T19:31:38.607 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 1 librados: calling monclient init 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient: init 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed00b1730 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed00b4d60 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed014be90 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed01dafb0 con 0x7f4ed014be90 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed013d0f0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed013d280 con 0x7f4ed00b1730 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:38.608 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.605+0000 7f4ed5c27640 10 monclient(hunting): authenticate will time out at 7025.704689s 2024-03-25T19:31:38.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.611+0000 7f4ec6ffd640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f4ed00b4d60 0x7f4ed01449b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:50210/0 (socket says 172.21.15.60:50210) 2024-03-25T19:31:38.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.611+0000 7f4ec6ffd640 1 -- 172.21.15.60:0/228171634 learned_addr learned my addr 172.21.15.60:0/228171634 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:38.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.612+0000 7f4eccff9640 10 client.?.objecter ms_handle_connect 0x7f4ed00b4d60 2024-03-25T19:31:38.612 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.612+0000 7f4eccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:38.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.618+0000 7f4eccff9640 10 client.?.objecter ms_handle_connect 0x7f4ed014be90 2024-03-25T19:31:38.618 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.618+0000 7f4eccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:38.621 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.621+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171062289 0 0) 0x7f4ed013d0f0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.621 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.621+0000 7f4eccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:38.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.621+0000 7f4eccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:38.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.621+0000 7f4eccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:38.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.621+0000 7f4eccff9640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:31:38.622 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.621+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0004eb0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.630 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.630+0000 7f4eccff9640 10 client.?.objecter ms_handle_connect 0x7f4ed00b1730 2024-03-25T19:31:38.630 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.630+0000 7f4eccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:38.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.633+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517941114 0 0) 0x7f4ed01dafb0 con 0x7f4ed014be90 2024-03-25T19:31:38.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.633+0000 7f4eccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:38.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.633+0000 7f4eccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:38.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.633+0000 7f4eccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:38.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.633+0000 7f4eccff9640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:31:38.634 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.633+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed013d0f0 con 0x7f4ed014be90 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.634+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897247122 0 0) 0x7f4eb0004eb0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.634+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ed01dafb0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.634+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4eb80019b0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.634+0000 7f4eccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.634+0000 7f4eccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.634+0000 7f4eccff9640 10 monclient(hunting): dump: 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:38.635 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:38.636 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:38.636 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:38.636 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:38.636 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:38.636 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.634+0000 7f4eccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:38.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.648+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259479706 0 0) 0x7f4ed013d280 con 0x7f4ed00b1730 2024-03-25T19:31:38.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.648+0000 7f4eccff9640 10 monclient(hunting): _negotiate 2024-03-25T19:31:38.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.648+0000 7f4eccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:38.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.648+0000 7f4eccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:38.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.648+0000 7f4eccff9640 10 monclient(hunting): my global_id is 4724 2024-03-25T19:31:38.648 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.648+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0004eb0 con 0x7f4ed00b1730 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3893577038 0 0) 0x7f4ed01dafb0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 >> v1:172.21.15.60:6790/0 conn(0x7f4ed00b1730 legacy=0x7f4ed01db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 >> v1:172.21.15.60:6789/0 conn(0x7f4ed014be90 legacy=0x7f4ed01e2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 10 monclient: _finish_hunting 0 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 1 monclient: found mon.b 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed01dbbe0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 10 monclient: _finish_auth 0 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 10 monclient: _check_auth_tickets 2024-03-25T19:31:38.656 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4eccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 10 monclient: _renew_subs 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed0143900 con 0x7f4ed00b4d60 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 10 monclient: _renew_subs 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed01452a0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.655+0000 7f4ed5c27640 1 librados: init done 2024-03-25T19:31:38.657 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.656+0000 7f4ed5c27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:38.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.656+0000 7f4ed5c27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:38.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.656+0000 7f4ed5c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:38.658 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.656+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed01db140 con 0x7f4ed00b4d60 2024-03-25T19:31:38.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.662+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4eb80047e0 con 0x7f4ed00b4d60 2024-03-25T19:31:38.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.662+0000 7f4eccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:38.663 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.662+0000 7f4ece7fc640 4 set_mon_vals no callback set 2024-03-25T19:31:38.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.942+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f4eb8004b30 con 0x7f4ed00b4d60 2024-03-25T19:31:38.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.942+0000 7f4eccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:38.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.942+0000 7f4eccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:38.943 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.942+0000 7f4eccff9640 10 monclient: dump: 2024-03-25T19:31:38.943 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:38.943 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:38.943 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:38.944 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.943+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f4eb8004d10 con 0x7f4ed00b4d60 2024-03-25T19:31:38.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.944+0000 7f4ecf7fe640 10 monclient: get_auth_request con 0x7f4eb006de00 auth_method 0 2024-03-25T19:31:38.945 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.944+0000 7f4eccff9640 10 client.4729.objecter ms_handle_connect 0x7f4eb006de00 2024-03-25T19:31:38.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.946+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f4eb8054910 con 0x7f4ed00b4d60 2024-03-25T19:31:38.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.946+0000 7f4eccff9640 10 client.4729.objecter ms_dispatch 0x7f4ed0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:38.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.946+0000 7f4eccff9640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:38.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.946+0000 7f4eccff9640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:38.947 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.946+0000 7f4eccff9640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T19:31:38.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.949+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4eb8050250 con 0x7f4ed00b4d60 2024-03-25T19:31:38.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.949+0000 7f4eccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:38.949 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:38.949+0000 7f4eccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:39.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.210+0000 7f4ed5c27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:39.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.210+0000 7f4ed5c27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:39.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.210+0000 7f4ed5c27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:39.211 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.210+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ed00627b0 con 0x7f4ed00b4d60 2024-03-25T19:31:39.243 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.243+0000 7f4eccff9640 1 -- 172.21.15.60:0/228171634 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f4eb8052350 con 0x7f4ed00b4d60 2024-03-25T19:31:39.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.243+0000 7f4eccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:39.244 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.243+0000 7f4eccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:39.244 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:31:39.244 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:39.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 10 librados: watch_flush enter 2024-03-25T19:31:39.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 10 librados: watch_flush exit 2024-03-25T19:31:39.246 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 10 client.4729.objecter successfully canceled tick 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 >> v1:172.21.15.90:6800/1827638754 conn(0x7f4eb006de00 legacy=0x7f4eb00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 10 monclient: shutdown 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 >> v1:172.21.15.90:6789/0 conn(0x7f4ed00b4d60 legacy=0x7f4ed01449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 shutdown_connections 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 >> 172.21.15.60:0/228171634 conn(0x7f4ed00617a0 msgr2=0x7f4ed00b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.245+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 shutdown_connections 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.246+0000 7f4ed5c27640 1 -- 172.21.15.60:0/228171634 wait complete. 2024-03-25T19:31:39.247 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:39.246+0000 7f4ed5c27640 1 librados: shutdown 2024-03-25T19:31:39.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:41.580 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:42.291 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.595 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:42.306 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.633 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:11:42.344 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.646 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:42.357 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.653 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:42.306 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.656 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:42.310 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.664 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:42.375 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.678 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:11:42.332 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:41.687 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:11:42.399 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:31:46.065 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.181+0000 7f3944df4640 10 monclient: get_monmap_and_config 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.181+0000 7f3944df4640 10 monclient: build_initial_monmap 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.181+0000 7f3944df4640 10 monclient: monmap: 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:31:46.182745+0000 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:31:46.182745+0000 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:46.182 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-c 2024-03-25T19:31:46.183 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-b 2024-03-25T19:31:46.183 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.181+0000 7f3944df4640 1 Processor -- start 2024-03-25T19:31:46.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.181+0000 7f3944df4640 1 -- start start 2024-03-25T19:31:46.183 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.181+0000 7f3944df4640 10 monclient: init 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 10 monclient(hunting): picked mon.noname-c con 0x7f39400a98f0 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 10 monclient(hunting): picked mon.noname-a con 0x7f394013f710 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 10 monclient(hunting): picked mon.noname-b con 0x7f39401430c0 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3940142520 con 0x7f394013f710 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b28a0 con 0x7f39400a98f0 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b2a30 con 0x7f39401430c0 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:46.184 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.182+0000 7f3944df4640 10 monclient(hunting): authenticate will time out at 7033.281585s 2024-03-25T19:31:46.194 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.193+0000 7f393e59c640 1 --1- >> v1:172.21.15.90:6789/0 conn(0x7f39400a98f0 0x7f39400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.90:6789/0 says I am v1:172.21.15.60:34078/0 (socket says 172.21.15.60:34078) 2024-03-25T19:31:46.194 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.193+0000 7f393e59c640 1 -- 172.21.15.60:0/4268208349 learned_addr learned my addr 172.21.15.60:0/4268208349 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:46.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.204+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606299880 0 0) 0x7f39400b28a0 con 0x7f39400a98f0 2024-03-25T19:31:46.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.204+0000 7f393dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:31:46.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.204+0000 7f393dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:46.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.204+0000 7f393dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:46.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.204+0000 7f393dd9b640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:31:46.204 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.204+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f392c003520 con 0x7f39400a98f0 2024-03-25T19:31:46.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.206+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098831951 0 0) 0x7f3940142520 con 0x7f394013f710 2024-03-25T19:31:46.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.206+0000 7f393dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:31:46.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.206+0000 7f393dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:46.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.206+0000 7f393dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:46.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.206+0000 7f393dd9b640 10 monclient(hunting): my global_id is 4746 2024-03-25T19:31:46.207 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.206+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39400b28a0 con 0x7f394013f710 2024-03-25T19:31:46.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.207+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421385581 0 0) 0x7f39400b2a30 con 0x7f39401430c0 2024-03-25T19:31:46.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.207+0000 7f393dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:31:46.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.207+0000 7f393dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:46.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.207+0000 7f393dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:46.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.207+0000 7f393dd9b640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:31:46.208 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.207+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940142520 con 0x7f39401430c0 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3519138931 0 0) 0x7f39400b28a0 con 0x7f394013f710 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 >> v1:172.21.15.60:6790/0 conn(0x7f39401430c0 legacy=0x7f394014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 >> v1:172.21.15.90:6789/0 conn(0x7f39400a98f0 legacy=0x7f39400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.60:6789/0 2024-03-25T19:31:46.217 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 --> v1:172.21.15.60:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39400b44c0 con 0x7f394013f710 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3928001670 con 0x7f394013f710 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f393dd9b640 10 monclient: dump: 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:46.218 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:46.219 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:46.219 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f3944df4640 5 monclient: authenticate success, global_id 4746 2024-03-25T19:31:46.219 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.216+0000 7f3944df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:31:46.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 <== mon.0 v1:172.21.15.60:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39280017d0 con 0x7f394013f710 2024-03-25T19:31:46.224 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f393dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f393dd9b640 1 -- 172.21.15.60:0/4268208349 <== mon.0 v1:172.21.15.60:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3928001be0 con 0x7f394013f710 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f3944df4640 10 monclient: get_monmap_and_config success 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f3944df4640 4 set_mon_vals no callback set 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f393dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f393dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.224+0000 7f393dd9b640 10 monclient: dump: 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:46.225 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:46.226 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.225+0000 7f3944df4640 10 monclient: shutdown 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.225+0000 7f3944df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.225+0000 7f3944df4640 1 -- 172.21.15.60:0/4268208349 >> v1:172.21.15.60:6789/0 conn(0x7f394013f710 legacy=0x7f39401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.225+0000 7f3944df4640 1 -- 172.21.15.60:0/4268208349 shutdown_connections 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.225+0000 7f3944df4640 1 -- 172.21.15.60:0/4268208349 >> 172.21.15.60:0/4268208349 conn(0x7f39400b7ea0 msgr2=0x7f39400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.225+0000 7f3944df4640 1 -- 172.21.15.60:0/4268208349 shutdown_connections 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.225+0000 7f3944df4640 1 -- 172.21.15.60:0/4268208349 wait complete. 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 10 monclient: build_initial_monmap 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 10 monclient: monmap: 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:epoch 0 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:last_changed 0.000000 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:created 0.000000 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 0 (unknown) 2024-03-25T19:31:46.228 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 1 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.noname-a 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.noname-b 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.noname-c 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 1 librados: starting msgr at 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 1 librados: starting objecter 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 1 Processor -- start 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 1 -- start start 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 1 librados: setting wanted keys 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 1 librados: calling monclient init 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.226+0000 7f3944df4640 10 monclient: init 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 10 monclient: _reopen_session rank -1 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 10 monclient(hunting): picked mon.noname-c con 0x7f39400a98f0 addr v1:172.21.15.60:6790/0 2024-03-25T19:31:46.229 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 10 monclient(hunting): picked mon.noname-b con 0x7f394013f710 addr v1:172.21.15.90:6789/0 2024-03-25T19:31:46.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 10 monclient(hunting): picked mon.noname-a con 0x7f39401430c0 addr v1:172.21.15.60:6789/0 2024-03-25T19:31:46.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 1 -- --> v1:172.21.15.60:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3940139190 con 0x7f39401430c0 2024-03-25T19:31:46.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 1 -- --> v1:172.21.15.90:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3940137590 con 0x7f394013f710 2024-03-25T19:31:46.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 1 -- --> v1:172.21.15.60:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3940138b40 con 0x7f39400a98f0 2024-03-25T19:31:46.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 10 monclient(hunting): _renew_subs 2024-03-25T19:31:46.230 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.227+0000 7f3944df4640 10 monclient(hunting): authenticate will time out at 7033.326448s 2024-03-25T19:31:46.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.231+0000 7f393ed9d640 1 --1- >> v1:172.21.15.60:6789/0 conn(0x7f39401430c0 0x7f39401e1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.60:6789/0 says I am v1:172.21.15.60:47174/0 (socket says 172.21.15.60:47174) 2024-03-25T19:31:46.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.231+0000 7f393ed9d640 1 -- 172.21.15.60:0/971604448 learned_addr learned my addr 172.21.15.60:0/971604448 (peer_addr_for_me v1:172.21.15.60:0/0) 2024-03-25T19:31:46.232 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.232+0000 7f39377fe640 10 client.?.objecter ms_handle_connect 0x7f39401430c0 2024-03-25T19:31:46.233 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.232+0000 7f39377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:46.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.244+0000 7f39377fe640 10 client.?.objecter ms_handle_connect 0x7f394013f710 2024-03-25T19:31:46.245 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.244+0000 7f39377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:46.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.250+0000 7f39377fe640 10 client.?.objecter ms_handle_connect 0x7f39400a98f0 2024-03-25T19:31:46.250 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.250+0000 7f39377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:31:46.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.251+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.0 v1:172.21.15.60:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709487559 0 0) 0x7f3940139190 con 0x7f39401430c0 2024-03-25T19:31:46.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.251+0000 7f39377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:46.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.251+0000 7f39377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:46.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.251+0000 7f39377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:46.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.251+0000 7f39377fe640 10 monclient(hunting): my global_id is 4749 2024-03-25T19:31:46.252 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.251+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.60:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3920004eb0 con 0x7f39401430c0 2024-03-25T19:31:46.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.256+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084525465 0 0) 0x7f3940137590 con 0x7f394013f710 2024-03-25T19:31:46.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.256+0000 7f39377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:46.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.256+0000 7f39377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:46.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.256+0000 7f39377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:46.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.256+0000 7f39377fe640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:31:46.257 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.256+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.90:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940139190 con 0x7f394013f710 2024-03-25T19:31:46.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.269+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.2 v1:172.21.15.60:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063563573 0 0) 0x7f3940138b40 con 0x7f39400a98f0 2024-03-25T19:31:46.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.269+0000 7f39377fe640 10 monclient(hunting): _negotiate 2024-03-25T19:31:46.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.269+0000 7f39377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:31:46.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.269+0000 7f39377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:31:46.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.269+0000 7f39377fe640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:31:46.270 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.269+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.60:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3940137590 con 0x7f39400a98f0 2024-03-25T19:31:46.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.277+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4269261069 0 0) 0x7f3940139190 con 0x7f394013f710 2024-03-25T19:31:46.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.277+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.90:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3940138b40 con 0x7f394013f710 2024-03-25T19:31:46.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.278+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3928001b60 con 0x7f394013f710 2024-03-25T19:31:46.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.278+0000 7f39377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:46.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.278+0000 7f39377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:31:46.278 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.278+0000 7f39377fe640 10 monclient(hunting): dump: 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.279 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.278+0000 7f39377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:31:46.285 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.285+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.2 v1:172.21.15.60:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 5978114 0 0) 0x7f3940137590 con 0x7f39400a98f0 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.285+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.60:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3940139190 con 0x7f39400a98f0 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.285+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.2 v1:172.21.15.60:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3930001b30 con 0x7f39400a98f0 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.285+0000 7f39377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.285+0000 7f39377fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.285+0000 7f39377fe640 10 monclient(hunting): dump: 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:46.286 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.286+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.0 v1:172.21.15.60:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1886667195 0 0) 0x7f3920004eb0 con 0x7f39401430c0 2024-03-25T19:31:46.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.286+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.60:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3940137590 con 0x7f39401430c0 2024-03-25T19:31:46.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.286+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.0 v1:172.21.15.60:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f39380018f0 con 0x7f39401430c0 2024-03-25T19:31:46.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.286+0000 7f39377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:46.287 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.286+0000 7f39377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.286+0000 7f39377fe640 10 monclient(hunting): dump: 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:46.288 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 759431619 0 0) 0x7f3940138b40 con 0x7f394013f710 2024-03-25T19:31:46.294 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 >> v1:172.21.15.60:6790/0 conn(0x7f39400a98f0 legacy=0x7f3940136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 >> v1:172.21.15.60:6789/0 conn(0x7f39401430c0 legacy=0x7f39401e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 10 monclient: _finish_hunting 0 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 1 monclient: found mon.b 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.90:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3940136bf0 con 0x7f394013f710 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 10 monclient: _finish_auth 0 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 10 monclient: _check_auth_tickets 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f39377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f3944df4640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f3944df4640 10 monclient: _renew_subs 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f3944df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.293+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.90:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394013afe0 con 0x7f394013f710 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.294+0000 7f3944df4640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:31:46.295 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.294+0000 7f3944df4640 10 monclient: _renew_subs 2024-03-25T19:31:46.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.294+0000 7f3944df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:46.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.294+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.90:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3940139620 con 0x7f394013f710 2024-03-25T19:31:46.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.294+0000 7f3944df4640 1 librados: init done 2024-03-25T19:31:46.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.295+0000 7f3944df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:46.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.295+0000 7f3944df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:46.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.295+0000 7f3944df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:46.296 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.295+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3940074cc0 con 0x7f394013f710 2024-03-25T19:31:46.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.579+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39280038a0 con 0x7f394013f710 2024-03-25T19:31:46.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.579+0000 7f39377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:31:46.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.579+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4180600922 0 0) 0x7f3928003bb0 con 0x7f394013f710 2024-03-25T19:31:46.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.579+0000 7f39377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:31:46.580 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.579+0000 7f39377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.579+0000 7f39377fe640 10 monclient: dump: 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:epoch 1 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:fsid 69068b48-8881-455b-b2ea-bfe0d20a1ddc 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:last_changed 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:created 2024-03-25T19:19:28.404051+0000 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:min_mon_release 19 (squid) 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:election_strategy: 3 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:0: v1:172.21.15.60:6789/0 mon.a 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:1: v1:172.21.15.90:6789/0 mon.b 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:2: v1:172.21.15.60:6790/0 mon.c 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr: 2024-03-25T19:31:46.581 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.579+0000 7f393d59a640 4 set_mon_vals no callback set 2024-03-25T19:31:46.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.580+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1254380678 0 0) 0x7f3928019b80 con 0x7f394013f710 2024-03-25T19:31:46.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.581+0000 7f393e59c640 10 monclient: get_auth_request con 0x7f392006e0c0 auth_method 0 2024-03-25T19:31:46.582 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.581+0000 7f39377fe640 10 client.4735.objecter ms_handle_connect 0x7f392006e0c0 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.585+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2158179258 0 0) 0x7f3928053690 con 0x7f394013f710 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.585+0000 7f39377fe640 10 client.4735.objecter ms_dispatch 0x7f3940000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.585+0000 7f39377fe640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.585+0000 7f39377fe640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.585+0000 7f39377fe640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.586+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f392804f1a0 con 0x7f394013f710 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.586+0000 7f39377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:31:46.586 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.586+0000 7f39377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:31:46.814 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.814+0000 7f3944df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:46.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.814+0000 7f3944df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:46.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.814+0000 7f3944df4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.90:6789/0 2024-03-25T19:31:46.815 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.814+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 --> v1:172.21.15.90:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3940138f60 con 0x7f394013f710 2024-03-25T19:31:46.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.861+0000 7f39377fe640 1 -- 172.21.15.60:0/971604448 <== mon.1 v1:172.21.15.90:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2512275157) 0x7f39280510d0 con 0x7f394013f710 2024-03-25T19:31:46.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.861+0000 7f39377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:31:46.862 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.861+0000 7f39377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:31:46.862 INFO:teuthology.misc.health.smithi060.stdout: 2024-03-25T19:31:46.862 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":11,"fsid":"69068b48-8881-455b-b2ea-bfe0d20a1ddc","created":"2024-03-25T19:20:16.689428+0000","modified":"2024-03-25T19:30:21.209954+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:30:21.209954+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"63dcc61d-0867-4284-a895-94136baa98f9","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"14a85c8d-cc53-4f18-8189-46f512c990db","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d670f672-5f2a-46b2-a7d5-d2957956e0c4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"98f1537b-88fb-42c5-b3b6-af8d02f925f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5444b0c8-c71c-4fa2-b901-c4660901c646","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae0a333a-7fb1-41ce-b567-a6322ab9b7a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4da3c427-3ae3-460c-afbb-b59f3303db7b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4f96d257-76d5-456f-9a43-180cc136ed55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 10 librados: watch_flush enter 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 10 librados: watch_flush exit 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 10 client.4735.objecter successfully canceled tick 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 >> v1:172.21.15.90:6800/1827638754 conn(0x7f392006e0c0 legacy=0x7f3920070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 10 monclient: shutdown 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 >> v1:172.21.15.90:6789/0 conn(0x7f394013f710 legacy=0x7f39401364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:31:46.865 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 shutdown_connections 2024-03-25T19:31:46.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 >> 172.21.15.60:0/971604448 conn(0x7f39400b7ea0 msgr2=0x7f3940074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:31:46.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 shutdown_connections 2024-03-25T19:31:46.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 1 -- 172.21.15.60:0/971604448 wait complete. 2024-03-25T19:31:46.866 INFO:teuthology.misc.health.smithi060.stderr:2024-03-25T19:31:46.864+0000 7f3944df4640 1 librados: shutdown 2024-03-25T19:31:46.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:31:46.876 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:31:46.877 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:31:46.877 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:31:46.877 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:48.816 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:11:26.746 35307== 2024-03-25T19:31:48.816 INFO:tasks.ceph.osd.0.smithi060.stderr:==00:00:11:26.746 35307== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:48.855 INFO:tasks.ceph.osd.0.smithi060.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:52.979 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:31:52.980 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:31:52.980 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:31:52.980 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:31:55.029 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:11:32.980 35306== 2024-03-25T19:31:55.030 INFO:tasks.ceph.osd.1.smithi060.stderr:==00:00:11:32.980 35306== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:31:55.073 INFO:tasks.ceph.osd.1.smithi060.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:31:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:31:59.082 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:31:59.082 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:31:59.082 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:31:59.083 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:00.901 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:11:38.832 35308== 2024-03-25T19:32:00.901 INFO:tasks.ceph.osd.2.smithi060.stderr:==00:00:11:38.832 35308== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:00.949 INFO:tasks.ceph.osd.2.smithi060.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:01.638 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:02.349 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.643 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:02.354 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.674 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:02.386 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.693 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:12:02.405 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.695 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:02.349 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.698 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:02.352 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.699 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:12:02.411 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.714 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:02.368 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:01.730 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:12:02.441 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:05.184 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:32:05.185 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:32:05.185 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:32:05.185 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:07.091 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:11:45.019 35309== 2024-03-25T19:32:07.092 INFO:tasks.ceph.osd.3.smithi060.stderr:==00:00:11:45.019 35309== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:07.131 INFO:tasks.ceph.osd.3.smithi060.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:11.287 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:32:11.287 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:32:11.288 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:32:11.288 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:13.352 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:11:51.298 35740== 2024-03-25T19:32:13.352 INFO:tasks.ceph.osd.4.smithi090.stderr:==00:00:11:51.298 35740== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:13.390 INFO:tasks.ceph.osd.4.smithi090.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:17.390 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:32:17.390 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:32:17.391 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:32:17.391 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:19.220 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:11:57.151 35742== 2024-03-25T19:32:19.220 INFO:tasks.ceph.osd.5.smithi090.stderr:==00:00:11:57.151 35742== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:19.255 INFO:tasks.ceph.osd.5.smithi090.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:21.767 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:22.478 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.815 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:22.527 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.839 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:22.550 34990== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.860 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:22.514 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.868 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:22.522 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.883 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:12:22.537 35194== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.888 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:12:22.600 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.893 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:12:22.605 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:21.909 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:12:22.621 34991== Warning: unimplemented fcntl command: 1036 2024-03-25T19:32:23.493 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:23.493 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:32:23.493 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:32:23.494 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:32:23.494 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:25.209 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:12:03.134 35743== 2024-03-25T19:32:25.210 INFO:tasks.ceph.osd.6.smithi090.stderr:==00:00:12:03.134 35743== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:25.249 INFO:tasks.ceph.osd.6.smithi090.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:29.595 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:29.596 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:32:29.596 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:32:29.596 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:32:29.597 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:31.022 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:12:08.962 35741== 2024-03-25T19:32:31.022 INFO:tasks.ceph.osd.7.smithi090.stderr:==00:00:12:08.962 35741== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:31.052 INFO:tasks.ceph.osd.7.smithi090.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:35.698 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:35.699 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:32:35.699 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:32:35.699 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:32:35.700 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:32:35.700 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:35.725 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:32:35.725 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:32:35.725 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:35.726 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:32:35.727+0000 7faf18745640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35192) UID: 0 2024-03-25T19:32:35.726 INFO:tasks.ceph.mgr.x.smithi090.stderr:2024-03-25T19:32:35.727+0000 7faf18745640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:32:35.752 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:32:35.752 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:32:35.752 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:32:35.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:35.781 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-03-25T19:32:35.770+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34956) UID: 0 2024-03-25T19:32:35.781 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-03-25T19:32:35.772+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:32:36.344 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:37.055 34990== 2024-03-25T19:32:36.344 INFO:tasks.ceph.mon.a.smithi060.stderr:==00:00:12:37.055 34990== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:32:36.353 INFO:tasks.ceph.mon.a.smithi060.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:32:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:32:41.854 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T19:32:41.854 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T19:32:41.855 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T19:32:41.855 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:32:41.871 INFO:tasks.ceph.mon.c.smithi060.stderr:2024-03-25T19:32:41.860+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34958) UID: 0 2024-03-25T19:32:41.872 INFO:tasks.ceph.mon.c.smithi060.stderr:2024-03-25T19:32:41.862+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:34:03.576 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:14:04.287 34991== 2024-03-25T19:34:03.576 INFO:tasks.ceph.mon.c.smithi060.stderr:==00:00:14:04.287 34991== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:03.586 INFO:tasks.ceph.mon.c.smithi060.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:07.277 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T19:34:07.278 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T19:34:07.279 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T19:34:07.279 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:34:07.296 INFO:tasks.ceph.mon.b.smithi090.stderr:2024-03-25T19:34:07.283+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35190) UID: 0 2024-03-25T19:34:07.297 INFO:tasks.ceph.mon.b.smithi090.stderr:2024-03-25T19:34:07.285+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T19:34:53.915 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:54.569 35194== 2024-03-25T19:34:53.916 INFO:tasks.ceph.mon.b.smithi090.stderr:==00:00:14:54.569 35194== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:34:53.925 INFO:tasks.ceph.mon.b.smithi090.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:34:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:34:56.090 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T19:34:56.091 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T19:34:56.091 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:34:56.091 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:34:56.142 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:34:56.142 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:34:56.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:34:56.197 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:34:56.270 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:34:56.270 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:34:56.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:34:56.330 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:34:56.387 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:34:56.387 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:34:56.492 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:34:56.492 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:34:56.555 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:34:56.555 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:34:56.629 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:34:56.630 DEBUG:teuthology.orchestra.run.smithi090:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:34:56.686 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:34:56.687 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/data/mon.a.tgz 2024-03-25T19:34:56.688 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-03-25T19:34:56.703 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.KVrlWJK9en 2024-03-25T19:34:56.703 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.KVrlWJK9en 2024-03-25T19:34:56.810 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.KVrlWJK9en 2024-03-25T19:34:56.897 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.KVrlWJK9en is 132KB 2024-03-25T19:34:56.948 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.KVrlWJK9en 2024-03-25T19:34:56.962 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/data/mon.c.tgz 2024-03-25T19:34:56.962 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-03-25T19:34:57.015 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.R5GhcEzMAW 2024-03-25T19:34:57.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.R5GhcEzMAW 2024-03-25T19:34:57.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.R5GhcEzMAW 2024-03-25T19:34:57.186 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.R5GhcEzMAW is 133KB 2024-03-25T19:34:57.237 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.R5GhcEzMAW 2024-03-25T19:34:57.251 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/data/mon.b.tgz 2024-03-25T19:34:57.251 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-03-25T19:34:57.268 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.LaylBqqhdQ 2024-03-25T19:34:57.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.LaylBqqhdQ 2024-03-25T19:34:57.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.LaylBqqhdQ 2024-03-25T19:34:57.458 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.LaylBqqhdQ is 133KB 2024-03-25T19:34:57.482 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.LaylBqqhdQ 2024-03-25T19:34:57.496 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:34:57.497 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:34:57.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:34:57.883 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:34:58.124 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:34:58.126 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/remote/smithi060/crash 2024-03-25T19:34:58.127 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:34:58.153 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/remote/smithi090/crash 2024-03-25T19:34:58.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:34:58.180 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:34:58.180 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:34:58.198 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:34:58.221 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2024-03-25T19:34:58.221 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41848.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35641.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35641.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35641.log.gz 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:34:58.271 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/mon.b.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35430.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35350.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35550.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35430.log.gz 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35390.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35350.log.gz 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35470.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35550.log.gz 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35510.log 2024-03-25T19:34:58.272 INFO:teuthology.orchestra.run.smithi090.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35470.log.gz 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.admin.35510.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35596.log 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35510.log.gz 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-client.admin.35596.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35596.log.gz 2024-03-25T19:34:58.273 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:34:58.275 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35476.log 2024-03-25T19:34:58.275 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41848.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:34:58.275 INFO:teuthology.orchestra.run.smithi060.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2024-03-25T19:34:58.275 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41965.log 2024-03-25T19:34:58.275 INFO:teuthology.orchestra.run.smithi060.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41848.log.gz 2024-03-25T19:34:58.275 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39427.log 2024-03-25T19:34:58.275 INFO:teuthology.orchestra.run.smithi060.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35476.log.gz 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39700.log 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39310.log 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41965.log.gz 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38998.log 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39427.log.gz 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41887.log 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39700.log.gz 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38998.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39310.log.gz 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38839.log 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41651.log 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38998.log.gz 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41887.log.gz 2024-03-25T19:34:58.276 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41651.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38839.log.gz 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41651.log.gz 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40168.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 87.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz -- 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr: /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:34:58.277 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39037.log 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41692.log 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40754.log 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41692.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39037.log.gz 2024-03-25T19:34:58.278 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41809.log 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41692.log.gz 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40871.log 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40754.log.gz 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39154.log 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41809.log.gz 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40871.log.gz 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41612.log 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39934.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39154.log.gz 2024-03-25T19:34:58.279 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41770.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41612.log.gz 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34998.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38958.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41770.log.gz 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.34998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/ceph-client.admin.38958.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34998.log.gz 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39661.log 2024-03-25T19:34:58.280 INFO:teuthology.orchestra.run.smithi060.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38958.log.gz 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42122.log 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39661.log.gz 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40637.log: /var/log/ceph/ceph-client.admin.40051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42083.log 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42122.log.gz 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40363.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-03-25T19:34:58.281 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39544.log 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41378.log 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42083.log.gz 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39544.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40832.log 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39544.log.gz 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42043.log 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40832.log: 85.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:34:58.282 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40832.log.gz 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38619.log 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42043.log.gz 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40402.log: /var/log/ceph/ceph-client.admin.38619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41456.log 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41261.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38619.log.gz 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41926.log 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41456.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41926.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41456.log.gz 2024-03-25T19:34:58.283 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38878.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35332.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39388.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41926.log.gz 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40324.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35332.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40676.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38878.log.gz 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35332.log.gz 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39973.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40324.log.gz 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40676.log.gz 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-25T19:34:58.284 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40715.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39973.log.gz -5 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41573.log 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40441.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29132.log 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41573.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.tmp-client.admin.29132.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29132.log.gz -5 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41573.log.gz 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39583.log 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.41144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41534.log 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-03-25T19:34:58.285 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39778.log 86.3% 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39583.log.gz 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37318.log 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41534.log.gz 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.37318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39778.log.gz 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40988.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37318.log.gz 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr: 86.6%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz --verbose 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41027.log: -- /var/log/ceph/ceph-client.admin.40910.log 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39232.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:34:58.286 INFO:teuthology.orchestra.run.smithi060.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39349.log 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mgr.y.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39193.log 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:gzip/var/log/ceph/ceph-client.admin.39349.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40285.log 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39193.log: gzip 86.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39349.log.gz 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39193.log.gz 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40090.log 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40285.log.gz 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-03-25T19:34:58.287 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40090.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40949.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40090.log.gz 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35305.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.0.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39505.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.35305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr: 86.0%/var/log/ceph/ceph-client.admin.39505.log: -- replaced with /var/log/ceph/ceph-client.admin.35305.log.gz 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39505.log.gz 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.40207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.2.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39817.log 2024-03-25T19:34:58.288 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40793.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39817.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42004.log 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39817.log.gz 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.40129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.42004.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42004.log.gz 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38919.log 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41300.log 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.38919.log: /var/log/ceph/ceph-client.admin.40519.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38919.log.gz 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:34:58.289 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39466.log 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39466.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41300.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36350.log 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- 86.7% /var/log/ceph/ceph-client.admin.39856.log 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39466.log.gz 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.41495.log: gzip/var/log/ceph/ceph-client.admin.36350.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39856.log: 87.6% 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36350.log.gz 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:gzip 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:34:58.290 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.39739.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2024-03-25T19:34:58.312 INFO:teuthology.orchestra.run.smithi090.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:34:59.091 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:34:59.105 INFO:teuthology.orchestra.run.smithi060.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:34:59.175 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:34:59.178 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:34:59.183 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:34:59.365 INFO:teuthology.orchestra.run.smithi060.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:34:59.389 INFO:teuthology.orchestra.run.smithi060.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:34:59.443 INFO:teuthology.orchestra.run.smithi090.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:35:03.394 INFO:teuthology.orchestra.run.smithi060.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:35:03.555 INFO:teuthology.orchestra.run.smithi090.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:35:03.557 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-03-25T19:35:03.557 INFO:teuthology.orchestra.run.smithi090.stderr:real 0m5.315s 2024-03-25T19:35:03.557 INFO:teuthology.orchestra.run.smithi090.stderr:user 0m8.983s 2024-03-25T19:35:03.557 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m0.341s 2024-03-25T19:35:03.638 INFO:teuthology.orchestra.run.smithi060.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:35:03.640 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-03-25T19:35:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m5.430s 2024-03-25T19:35:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m14.119s 2024-03-25T19:35:03.640 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m0.525s 2024-03-25T19:35:03.640 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:35:03.640 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/remote/smithi060/log 2024-03-25T19:35:03.642 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:35:05.893 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/remote/smithi090/log 2024-03-25T19:35:05.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:35:07.352 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:35:07.562 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=77e3835612344ec4bc54bac9e9ec2501 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:35:07.566 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:35:07.576 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:35:07.594 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:35:07.595 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:35:07.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:35:07.689 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:35:07.700 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:35:07.700 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:35:07.703 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:35:07.717 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:35:07.720 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:35:07.721 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:35:07.721 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-03-25T19:35:07.721 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 -202us[ -222us] +/- 79ms 2024-03-25T19:35:07.722 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -350us[ -369us] +/- 49ms 2024-03-25T19:35:07.722 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +759us[ +739us] +/- 84ms 2024-03-25T19:35:07.722 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:35:07.724 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:35:07.724 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-03-25T19:35:07.724 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 -188us[ -200us] +/- 78ms 2024-03-25T19:35:07.725 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -335us[ -347us] +/- 49ms 2024-03-25T19:35:07.725 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +775us[ +775us] +/- 84ms 2024-03-25T19:35:07.725 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:35:07.726 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:35:07.734 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:35:07.735 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:35:07.745 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:35:07.836 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:35:07.907 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:35:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:35:07.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:35:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:35:07.971 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:35:07.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:35:07.989 INFO:teuthology.task.internal:Duration was 6857.058093 seconds 2024-03-25T19:35:07.989 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:35:08.002 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:35:08.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:35:08.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:35:08.047 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:35:08.058 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:35:08.434 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:35:08.434 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-03-25T19:35:08.435 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:35:08.456 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-03-25T19:35:08.456 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:35:08.483 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:35:08.483 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:35:08.498 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:35:08.554 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:35:08.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:35:08.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:35:08.702 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:35:08.712 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:35:08.712 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:35:08.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:35:08.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:35:08.776 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:35:08.781 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:35:08.803 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-03-25T19:35:08.832 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-03-25T19:35:08.858 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:35:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:35:08.881 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:35:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:35:08.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:35:08.921 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:35:08.922 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/remote/smithi060 2024-03-25T19:35:08.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:35:08.958 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299/remote/smithi090 2024-03-25T19:35:08.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:35:08.992 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:35:08.993 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:35:08.995 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:35:09.042 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:35:09.053 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:35:09.053 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:35:09.065 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:35:09.065 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:35:09.067 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:35:09.079 INFO:teuthology.orchestra.run.smithi060.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:35 /home/ubuntu/cephtest 2024-03-25T19:35:09.097 INFO:teuthology.orchestra.run.smithi090.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:35 /home/ubuntu/cephtest 2024-03-25T19:35:09.098 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:35:09.110 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:35:09.132 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:35:09.155 DEBUG:teuthology.nuke:shortname: smithi060 2024-03-25T19:35:09.155 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:35:09.177 DEBUG:teuthology.nuke:shortname: smithi090 2024-03-25T19:35:09.177 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:35:09.181 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:45.898802', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC8ogJIpy1TBt+IUhnJ7ZyvgSe3upjMvybdFt9GMEHJsjODab+wJCwQcVzxQfKN8/0bekXnyckYIX/8utXEO7xo='} 2024-03-25T19:35:09.201 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621299', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 17:30:45.897102', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdJt5xwrd3mWp6ij5wkj+fiGIV0KjKbkFyUqxxutZYyHZkvAiIrk7XMRh7jrWfGA9SGzmoXz15JMe8waoLYMJY='} 2024-03-25T19:35:09.235 INFO:teuthology.orchestra.console.smithi060:Power off 2024-03-25T19:35:09.236 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:35:09.252 INFO:teuthology.orchestra.console.smithi090:Power off 2024-03-25T19:35:09.252 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:35:09.262 DEBUG:teuthology.orchestra.console.smithi060:power off output: Chassis Power Control: Down/Off 2024-03-25T19:35:09.262 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:35:09.276 DEBUG:teuthology.orchestra.console.smithi090:power off output: Chassis Power Control: Down/Off 2024-03-25T19:35:09.277 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:35:09.288 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-03-25T19:35:09.300 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-25T19:35:13.289 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:35:13.301 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:35:13.401 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is on 2024-03-25T19:35:13.411 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-25T19:35:17.405 DEBUG:teuthology.orchestra.console.smithi060:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:35:17.414 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:35:17.516 DEBUG:teuthology.orchestra.console.smithi060:check power output: Chassis Power is off 2024-03-25T19:35:17.524 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is on 2024-03-25T19:35:17.618 INFO:teuthology.orchestra.console.smithi060:Power off completed 2024-03-25T19:35:21.529 DEBUG:teuthology.orchestra.console.smithi090:pexpect command: ipmitool -H smithi090.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:35:21.639 DEBUG:teuthology.orchestra.console.smithi090:check power output: Chassis Power is off 2024-03-25T19:35:21.739 INFO:teuthology.orchestra.console.smithi090:Power off completed 2024-03-25T19:35:21.844 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6857.058093070984 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=77e3835612344ec4bc54bac9e9ec2501 status: fail success: false 2024-03-25T19:35:21.844 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:35:21.924 INFO:teuthology.run:FAIL